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Executive Summary 

Magnet schools hold a prominent place in the history of education reforms in the United States. 
Best known for offering unique programs or curricula to attract students from outside a school’s 
neighborhood, many magnet schools started off as neighborhood public schools but converted 
with the goals of increasing student diversity and achievement. These goals remain important to 
policymakers and educators today, so there is interest in understanding what happens to 
converting schools, including those funded under the U.S. Department of Education’s Magnet 
School Assistance Program (MSAP). 

This report describes the results of a descriptive study of 21 MSAP-supported elementary 
schools. The study collected data on these schools for several years before and after their magnet 
conversion, to see how their student body composition and academic achievement changed over 
time. The group of schools contained 17 that converted to become what might be called 
“traditional” magnet schools and another 4 that converted to become “destination” magnet 
schools. 


Traditional Magnet Schools: Typically begin as lower performing schools serving higher 
proportions of students from low-income households or minority racial/ethnic groups than their 
districts; they are expected to recruit students who are higher achieving and more economically 
advantaged than the neighborhood students, or more likely to help the school achieve racial/ethnic 
diversity. 

Destination Magnet Schools: Typically high-performing schools serving higher proportions of 
economically advantaged or nonminority students; they are converted to magnets to serve as a 
destination for students from outside the neighborhood, who frequently attend sfruggling schools, are 
lower achieving, more economically disadvantaged, and more likely to be from minority racial/ethnic 
groups than are the neighborhood students. 


Key findings on the schools using the two conversion approaches include: 

■ When measured against district changes, both types of magnet schools experienced 
some changes in diversity in the expected direction. Over the conversion period 
(i.e., from before to after conversion) the traditional magnet schools reduced the 
concentration of students from minority racial/ethnic backgrounds relative to their 
districts, but there was no change in the concentration of economically disadvantaged 
students. Conversely, destination magnet schools increased their concentration of 
economically disadvantaged students relative to their districts, but had no change in the 
concentration of racial/ethnic minority students relative to their districts. Changes in 
diversity in the converting magnet schools were measured “relative to their districts” 
because the main way a school can change its composition is to attract students attending 
other schools in the district. 

■ Achievement in the traditional magnet schools was higher after conversion, 
outpacing district changes in English language arts (ELA) hut not in mathematics; 
achievement in destination magnet schools did not change, while their districts 
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improved over the conversion period. The changes in average student achievement 
could be due to increases (traditional magnet schools) or decreases (destination magnet 
schools) in learning by students who were in the schools both before and after 
conversion. But they could also be due to shifts in the types of students — higher 
achieving (traditional magnet schools) or lower achieving (destination magnet schools) — 
who came to the schools after conversion. As an example, if the economically 
disadvantaged students who were recruited to the destination magnet schools after 
conversion were lower achieving than the students who would normally attend those 
schools, the new students would lower average test scores in these just-established 
magnet schools. 

■ There is not evidence that magnet conversion itself played a role in the study 
schools’ diversity or achievement, with the exception of the decline in the 
concentration of minority students in traditional magnet schools. To assess whether 
the observed changes might have been produced by magnet conversion, changes in the 
magnet schools were compared to what would be predicted had they not converted. 
Predictions were made based on changes in neighborhood elementary schools in their 
districts that did not convert. If the changes were similar to what would be predicted, it 
would be harder to make the case that conversion was a factor in the magnet school 
changes. For almost all of the outcomes examined (minority racial/ethnic composition, 
disadvantaged student composition, and achievement), there was no significant difference 
between the changes in the magnet schools and the changes that would be predicted if 
they had not converted, except for the decrease in the minority student concentration in 
the traditional magnet schools relative to their districts. This promising result for the 
traditional magnet schools is not conclusive, however, because other factors (not just 
those that could affect neighborhood schools) cannot be ruled out. 

The Key Features of Magnet Schools and Theory of How They Might 
Improve Outcomes 

The key features of a magnet school are a specialized curriculum (e.g., performing arts or 
mathematics and science programs) or instructional method (e.g., open classrooms or team 
teaching), and enrollment that is open to students from outside the school neighborhood or 
attendance zone. The specialization in curriculum or instructional method is hypothesized to 
affect a school in two ways (Steel and Levine 1994). First, it is a vehicle to change the makeup 
of the student body, by attracting students from outside the school’s neighborhood who are 
different from students already attending the magnet school — in terms of their race/ethnicity, 
socioeconomic status, or academic achievement. Second, the specialized curriculum or teaching 
method can provide a focus for teacher professional development and collaboration and therefore 
improve the learning environment, in turn boosting student achievement. 

However, the specific paths by which this theory of action might result in increased student 
diversity and increased achievement depend on the characteristics of the neighborhood school 
before it converts and the types of students it is likely to recruit. As noted earlier, there are two 
common models of conversion to whole-school magnets supported through MSAP grant funds — 
the “traditional” approach and the “destination” approach. Both types were included in the study, 
though their outcomes were assessed separately. 
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How the Study Examined Change in Magnet Schoois 

The study relied mostly on student administrative records collected by and for the district and 
state. These records contained data on student characteristics, academic achievement, school 
assigned based on neighborhood of residence, and school attended for all elementary students in 
each district, typically for the two years before conversion and the four years after conversion. 
However, because of differences in when and how state academic assessments were conducted, 
the number of years and grades varied by district. To take the difference in tests into account, all 
scores were put into a common metric; in essence, each student’s mathematics and ELA score 
was standardized to show how well he or she performed relative to students in the same grade 
across the district that year. 

This study examined the changes that occurred in the 21 magnet schools from pre- to post- 
conversion. There are several limitations to the study’s approach. First, the design and methods 
of the study cannot measure the impact of the conversion to a magnet school, and can only 
provide suggestive, but not conclusive, evidence that conversion may have played a role in the 
changes in diversity or achievement that the magnet school attained. Second, although some 
MSAP converting schools may have set goals for improving diversity for specific racial/ethnic 
groups (e.g., increasing the proportion of Latino students but not African-American students), the 
study grouped all students from minority racial/ethnic backgrounds together. Thus, the study 
might have examined some changes in diversity that were not a focus of the schools’ efforts. 


Research Questions and Methods 

This study addressed three questions: 

Did the composition of neighborhood students and students from outside the 
neighborhood in the magnet schoois change after conversion? 

Both traditional and destination magnet schools were expected to attract more students from outside the 
neighborhood after conversion, and the outside students were expected to be different from the 
neighborhood students on various characteristics. To assess whether this happened, the proportion of 
students from outside the neighborhood in the study schools before conversion was compared to the 
proportion afterwards. Then the characteristics of students from outside the neighborhood were 
compared to the characteristics of neighborhood students — the proportion from racial/ethnic minority 
groups, the proportion who were economically disadvantaged (i.e., eligible for the federal free and 
reduced-price lunch program), and the average standardized test scores after conversion. Finally, since 
there were students from outside the neighborhood already in the magnet schools prior to conversion, an 
additional step examined whether the differences between the students from outside the neighborhood 
and the neighborhood students were greater after conversion. 

Did the diversity in the magnet schoois increase after conversion, and is there any 
evidence that it was reiated to conversion? 

The study focused on diversity with respect to the proportion of minority students and students from 
disadvantaged backgrounds. The analysis proceeded in two steps. 

■ Describing changes in magnets and their districts. As a first step, the pre- to postconversion 
change in magnet schools was compared with the change in the districts (including all elementary 
public schools) in which they were located. The goal of this analysis was to determine whether 
magnet schools changed in the anticipated direction relative to any overall change in their 
districts, and if so, how large the changes were. 
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■ Investigating the role of magnet conversion. After comparing the change in diversity in magnet 
schools to the change in their districts, a second analysis investigated whether magnet conversion 
might have played a role in any changes in diversity in magnet schools. To rule out other factors 
that might explain the observed change, the changes in magnet school student composition were 
compared to what would be predicted based on the changes in neighborhood schools that did not 
convert (i.e., schools that primarily served students within their neighborhood, excluding other 
magnets and charters). In this analysis, the change in other neighborhood schools represents a 
“counterfactual” — in other words, the change in other neighborhood schools represents what 
would have changed in magnet schools if they had not converted. (This is known as comparative 
interrupted time series analysis, and the methods are described in Appendix F.2.2.) 

Did the achievement in the magnet schoois increase after conversion, and is there any 
evidence that this was reiated to conversion? 

A primary goal of magnet conversion is to improve the achievement in the schools. As in the analysis of 
diversity, the analysis proceeded in two steps. 

■ Describing changes in magnets and their districts. First, the study examined whether the average 
mathematics and English language arts (ELA) achievement in the magnet schools was higher after 
conversion, by comparing the test scores of students attending after conversion to the test scores 
of students who were in those same grades in the years before conversion. 

■ Investigating the role of magnet conversion. To examine whether conversion may have played a 
role in any observed changes in achievement, the year-to-year achievement gains of students from 
the neighborhood before and after conversion were compared to what would be predicted based 
on the gains of similar neighborhood students in neighborhood schools that did not convert. (The 
methods used for this comparative intermpted time series analysis are described in Appendix F.4.2.) 
This analysis focused only on the traditional magnets for two reasons. Their neighborhood student 
populations — typically higher minority, more economically disadvantaged, and lower achieving — 
are of the greatest policy interest. In destination schools, the policy interest is the lower 
performing students from outside the neighborhood, and there are better approaches to examine 
the outcomes of students who transfer into schools, including methods based on lotteries. 

However, there were too few students from outside of the neighborhood transferring to destination 
magnets to apply these methods reliably. 
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Outcomes for Conversion Magnet Schools 

Most magnet school principals reported that they had fully implemented major components of 
their magnet programs by the third year of the grant (over 90 percent) and that they had 
experienced little problem building staff support (69 percent). Less than a third of MSAP 
directors reported that they experienced student recruiting difficulties due to competition from 
other choice schools (30 percent). The key issues related to outcomes are whether the converting 
magnet schools attained the improvements in diversity and achievement that motivates the 
MSAP program and whether their outcomes changed in ways that mirror — or set them apart 
from — neighborhood public schools in their districts that did not convert. 

Traditional Conversion Magnet Schools 

The traditional magnet schools served a somewhat larger share of students from 
outside their neighborhood after conversion, but these students were similar to 
the neighborhood students on a variety of characteristics. 

The share of students who were from outside the neighborhood rose by 5.8 percentage points 
(from 21.0 percent before conversion to 26.8 percent afterward), which equates to about one 
additional student in each class. However, after conversion, proportionately more of the outside 
students were economically disadvantaged — not advantaged, as would be expected based on the 
theory of action (an increase from 65.4 percent of outside students to 71.2 percent). Either the 
outside students who were at the schools before conversion became poorer or those the magnet 
schools recruited after conversion were poorer, or both. On average, after conversion, similar 
shares of students from inside and outside the neighborhood were from minority racial/ethnic 
groups (84.0 and 84.4 percent, respectively) or economically disadvantaged (73.8 percent and 
71.2 percent, respectively). The achievement of both student groups was also similar after 
conversion (about the 44th percentile in ELA and 48th percentile in mathematics for both student 
groups). Therefore, the neighborhood students were not exposed to higher achieving peers after 
conversion. 

Relative to their districts, there was less concentration of students from minority 
racial/ethnic backgrounds in the magnet schools after conversion, and the 
conversion may have played a role. 

Describing changes in magnet schools and their districts. The schools that became traditional 
magnet schools initially served a higher proportion of minority students (84.5 percent on 
average) relative to their districts (64.1 percent; Exhibit ES.l). But while the proportion of 
minority students enrolled in the converting schools stayed the same after conversion, the 
proportion in the district grew by about 2 percentage points (from 64.1 percent to 66.3 percent). 
These changes brought the composition of the magnet schools more in line with that of their 
districts, reducing the concentration of minority students in the magnet schools relative to their 
districts. 

Investigating the role of magnet conversion. Results indicate that conversion may have helped 
close the racial/ethnic gap between the schools and their districts, as expected for this type of 
conversion approach. After conversion, the raciahethnic composition of magnet schools was more 
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like their districts than the predicted composition based on neighborhood schools that did not 
convert. 


Exhibit ES.1. Concentration of Racial/Ethnic Minority Students in Traditional Magnet 
Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of minority students In the magnet schools (84.5 percent) was 

20.4 percentage points higher than the proportion In their districts (64.1 percent). From pre- to post-conversion, there was no 
statistically significant change in the magnet schools’ proportion (0.4 percentage points), while the proportion in their districts 
increased by 2.3 percentage points. As a result, after (post-) conversion, the magnet schools’ average proportion (84.9 percent) was 

18.5 percentage points higher than the proportion in their districts (66.3 percent) — a significant change in the difference between the 
magnet schools and their districts of -1 .9 percentage points. This change represents a reduction in the concentration of minority 
students in the magnet schools relative to their districts. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = M schools in 10 districts. Results include all 
students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit 1.4 for additional detail. 

SOURCE: District administrative data. 

Relative to their districts, the concentration of economically disadvantaged 
students in the traditional magnet schools was unchanged after conversion. 

Describing changes in magnet schools and their districts. The converting schools initially 
served a higher proportion of disadvantaged students (7 1 .4 percent) relative to their districts 
(46.1 percent; Exhibit ES.2). Over the conversion period, the magnet schools and their districts 
experienced similar increases in the share of their students who were economically 
disadvantaged (2.8 and 3.9 percentage points, respectively). Thus there was no change in the 
concentration of economically disadvantaged students in the magnet schools relative to their 
districts. 

Investigating the role of magnet conversion. Results suggest that conversion was not related to 
changes in the socioeconomic gap between the magnet schools and their districts. The 
socioeconomic composition of magnet schools was not closer to their districts after conversion 
than the predicted composition based on neighborhood schools that did not convert. 
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Exhibit ES.2. Concentration of Economically Disadvantaged Students in Traditional 
Magnet Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of economically disadvantaged students In the magnet schools 
(71 .4 percent) was 25.3 percentage points higher than the proportion In their districts (46.1 percent). From pre- to post-conversion, 
the average proportion of disadvantaged students In the magnet schools and their districts Increased by similar amounts (2.8 and 
3.9 percentage points, respectively). As a result, after (post-) conversion, the magnet schools' average proportion (74.2 percent) 
was 24.1 percentage points higher than the proportion in their districts (50.1 percent) — a non-significant change in the difference 
between the magnet schools and their districts of -1 .2 percentage points. While the magnet schools were 1 .2 percentage points 
closer to their districts after conversion, this change is not statistically different from zero, and thus does not represent a reduction in 
the concentration of disadvantaged students in the magnet schools. 

NOTE: See Appendix F for methods used for calculations and significance tests. A/ = 1 1 schools in seven districts. Results include 
all students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit 1.8 for additional detail. 

SOURCE: District administrative data. 


Achievement in the traditionai magnet schoois improved after conversion, 
outpacing district improvement in ELA but not in mathematics; however, there is 
not evidence that the conversion piayed a roie in improving the achievement 
gains of neighborhood students, the popuiation of poiicy interest. 

Describing changes in magnet schools and their districts. ELA achievement in both the 
eonverting magnet schools and their distriets overall improved after conversion, with the 
increase for the magnet schools greater by 2.5 percentile points (Exhibit ES.3). Magnet schools’ 
mathematics achievement also improved by 2.0 percentile points more than in the magnet 
schools’ districts, though the average mathematies test seore increases in the magnet sehools and 
their distriets are eonsidered statistieally similar. 

Investigating the role of magnet conversion. Results give little reason to conclude that 
conversion was a factor in the change in achievement in magnet schools. The changes in annual 
aehievement gains for neighborhood students in magnet schools were similar to what would be 
predicted had the sehools not converted. 
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Exhibit ES.3. Achievement in Traditional Magnet Schools and Their Districts (Average 
Across Schools) 
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EXHIBIT READS: The average ELA achievement In the magnet schools Increased by 8.1 percentile points from before (pre-) 
conversion (35.5 percentile) to after (post-) conversion (43.6 percentile). The average ELA achievement In their districts 
Increased by 5.6 percentile points from pre-conversion (51.1 percentile) to post-conversion (56.8 percentile). Therefore, the magnet 
schools Increased 2.5 percentile points more than their districts (an 8.1 percentile point Increase In the magnet schools compared to a 
5.7 percentile point Increase In their districts) — a statistically significant change. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = 17 schools In 10 districts. Differences were 
calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix 
Exhibit 1.12 for additional detail. 

SOURCE: District administrative data. 
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Destination Conversion Magnet Schoois 

The destination magnet schoois aiso served a somewhat iarger share of students 
from outside their neighborhood after conversion, and these students were 
different from the neighborhood students in ways consistent with theory. 

The share of students who were from outside the neighborhood rose by 7.3 percentage points 
(from 34.2 percent before conversion to 41.4 percent afterward). Before conversion, the students 
attending the schools from outside the neighborhood were similar to the neighborhood students 
in terms of race/ethnicity. After conversion outside students were more likely than neighborhood 
students to be members of minority racial/ethnic groups (by 2.5 percentage points) and 
economically disadvantaged (by 5.9 percentage points). Similarly, before conversion the 
academic achievement of students from inside and outside the neighborhood was similar, but 
afterwards the academic achievement of outside students was 5.0 percentile points lower than that 
of neighborhood students in mathematics (but similar in ELA). Together these changes suggest 
that the destination magnet schools attracted the minority, economically disadvantaged, and lower 
achieving students that they were theorized to attract, giving these students from outside the 
neighborhood exposure to higher achieving and more economically advantaged students. 

Reiative to their districts, the concentration of minority students in destination 
magnet schoois was unchanged after conversion. 

Describing changes in magnet schools and their districts. The schools that became destination 
magnet schools initially served the same share of students from minority racial/ethnic groups as 
their districts overall (71.6 percent and 72.0 percent, respectively; Exhibit ES.4). The proportion 
of these students enrolled in both the converting schools and their districts grew similarly after 
the conversion (by 3.3 and 1.8 percentage points, respectively), so there was no change in 
minority-student concentration in the magnet schools relative to their districts. 

Investigating the role of magnet conversion. The results indicate that conversion was not related 
to changes in the racial/ethnic gap between the magnet schools and their districts. The changes in 
magnet schools were similar to what would be predicted had the schools not converted. 
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Exhibit ES.4. Concentration of Racial/Ethnic Minority Students in the Destination Magnet 
Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of minority students In the magnet schools (71.6 percent) was 
0.4 percentage points lower than the proportion In their districts (72.0 percent). From pre- to post-conversion, there was a 
statistically significant Increase In the magnet schools' proportion (3.3 percentage points) and In the proportion In their districts 
(1.8 percentage points). As a result, after (post-) conversion, the magnet schools’ average proportion (74.9 percent) was 
significantly higher (1.1 percentage points) than the proportion in their districts (73.8 percent), but the change in the difference 
between the magnet schools and their districts (0.7 percentage points) was not significant. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Results include 
all students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit J.4 for additional detail. 

SOURCE: District administrative data. 

Relative to their districts, the concentration of economically disadvantaged 
students in the destination magnet schools was higher after conversion, as might 
be expected, but there is not evidence that the conversion was a factor. 

Describing changes in magnet schools and their districts. The schools that became destination 
magnet schools initially served a smaller proportion of disadvantaged students (25.0 percent 
on average) than their districts (38.1 percent; Exhibit ES.5). Although both the magnet schools 
and their districts overall served a larger share of these students after conversion, the increase 
was greater for the converting magnet schools (6.8 percentage points) than for their districts 
(3.0 percentage points). These changes brought the composition of the magnet schools more in 
line with that of their districts, or, in other words, increased the concentration of disadvantaged 
students in the magnet schools relative to their districts. This result is consistent with the theory 
of action for destination schools. 

Investigating the role of magnet conversion. The share of disadvantaged students in magnet 
schools was predicted to become less like their districts overall after conversion, based on 
changes in neighborhood schools that did not convert; in fact, they became more like their 
districts. These opposite trajectories created a difference between the destination magnet and 
what would be predicted had they not converted that was just short of being statistically reliable 
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according to standard benchmarks.' This suggests that something other than distrietwide factors 
might have played a role, and the conversion to a magnet school is one hypothesis for that result. 


Exhibit ES.5. Concentration of Economically Disadvantaged Students in the Destination 
Magnet Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of economically disadvantaged students In the magnet schools 
(25.0 percent) was 13.1 percentage points lower than the proportion In their districts (38.1 percent). From pre- to post-conversion, 
the average proportion of disadvantaged students Increased significantly In the magnet schools (6.8 percentage points) and their 
districts (3.0 percentage points). As a result, after (post-) conversion, the magnet schools’ average proportion (31.8 percent) was 
9.3 percentage points lower than the proportion In their districts (41 .1 percent), which was a significant change In the difference 
between the magnet schools and their districts of -3.8 percentage points. This change represents a significant increase in the 
concentration of disadvantaged students in the magnet schools relative to their districts. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Results include 
all students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit J.8 for additional detail. 

SOURCE: District administrative data. 


Achievement in destination magnet schools was unchanged after conversion, whereas 
achievement in their districts improved. 

Describing changes in magnet schools and their districts. The magnet sehools started out 
higher achieving in ELA than their districts (59th percentile compared to 52nd percentile; 
Exhibit ES.6) and their districts improved by 5.5 percentile points more than the magnet 
schools. The magnet schools also started out higher achieving in mathematics (58th compared 
to 52nd percentile) and the districts improved by 5.1 percentile points more than the magnet 
schools, but this difference was just short of being statistically reliable. 


' The difference between the actual change in the destination magnets and the change predicted based on that of the 
neighborhood schools has a /»-value of .066, just larger than a />- value of less than 0.05 that would be needed to be 
statistically realiable at convential levels. 
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Exhibit ES.6. Achievement in Destination Magnet Schools and Their Districts (Average 
Across Schools) 
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EXHIBIT READS: The average ELA achievement In the magnet schools did not significantly change (Increase of 1.4 percentile 
points) from before (pre-) conversion (58.8th percentile) to after (post-) conversion (60.2nd percentile). The average ELA 
achievement in their districts increased significantly, by 6.9 percentile points from preconversion (51 .6th percentile) to 
postconversion (58.5th percentile). Therefore, the districts increased 5.5 percentile points more than the magnet schools 
(a 1 .4 percentile point increase in the magnet schools compared to a 6.9 percentile point increase in their districts) — a statistically 
significant change. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Differences were 
calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix 
Exhibit J.12 for additional detail. 

SOURCE: District administrative data. 
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Chapter 1: Introduction 

Magnet schools hold a prominent place in the history of education reforms in the United States. 
Best known for offering unique programs or curricula to attract students from outside a school’s 
neighborhood, many magnet schools started off as regular neighborhood public schools but 
converted with the goals of increasing student diversity and achievement. Because these goals 
remain important to policymakers and educators today, interest continues in whether conversion 
magnet schools have been able to achieve these goals. This descriptive report describes the 
results of a study of 21 elementary schools supported through the Department of Education’s 
(ED’s) Magnet Schools Assistance Program (MSAP), following them for several years after their 
magnet conversion to see how their student body composition and academic achievement 
changed over time. 

Magnet Schools and the MSAP Program 

Magnet schools first emerged in the early 20th century, with districts establishing competitive- 
admission magnet schools to provide a rigorous curriculum for their highest achieving students 
(Einn and Hockett 2012; Steel and Levine 1994).^ Although this tradition still exists, many more 
magnet schools were established by districts in the 1960s and 1970s to promote desegregation 
and encourage parents to keep their children in the district’s public schools rather than placing 
them in private schools or moving to the suburbs (Blank, Levine, and Steel 1996). More recently, 
magnet schools have become a way for districts to provide school choice. Magnet schools are 
one of the many options provided to parents so that they can select a school to meet their 
children’s educational needs and interests. Current estimates suggest that there are about 
2,700 magnet schools across the United States, which is less than 3 percent of all public schools 
(Keaton 2012).^ 

Eederal support for magnet schools dates to the 1970s, when Congress authorized funds under the 
Elementary and Secondary Education Act (ESEA) to support school districts attempting to 
desegregate, including the establishment of magnet programs (Steel and Levine 1994). After a 
short gap in federal grants for magnet school implementation. Congress established the current 
MSAP in 1984 specifically to fund magnet schools “designed to bring students from different 
social, economic, ethnic and racial backgrounds together” (34 C.E.R. § 280.1 [2000]) and improve 
educational quality (see also Steel and Levine 1994). 

Today, the MSAP program continues to support the development of magnet school programs and 
has three main goals; 

■ Promoting Diversity, MSAP funds are intended to improve diversity by changing the 
mix of students attending schools receiving MSAP support. Historically, diversity in 


^ Early examples inelude San Franeiseo’s Lowell Fligh Sehool, Boston Latin Sehool, Chieago’s Lane Teeh, and the 
Bronx Fligh Sehool of Seienee. 

^ The number of magnet sehools reported in Keaton (2012) eomes from the Common Core of Data (CCD), whieh 
reports information from states that identifies whieh sehools are magnet sehools. 
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terms of race and ethnicity was emphasized over that of socioeconomic status, but the 
characteristics and measures used to assess schools’ success in meeting the diversity goal 
have broadened with legal challenges to using race-conscious assignment for students."^ 

In conjunction with this shift, in the 2004 grant year (from which some study schools 
were drawn), MSAP began to emphasize race-neutral language in describing objectives 
for promoting diversity. 

■ Enhancing Achievement. All MSAP magnet schools are expected to undertake 
curricular and instructional reforms that substantially bolster students’ academic 
achievement and career, technological, and professional skills. 

■ Expanding Choice. Beginning in the 2004 grant cycle, MSAP included a third goal: 
expanding school options within districts. In particular, MSAP grantee districts were to 
focus on expanding public school choice to students who attend low-performing schools 
(Office of Innovation and Improvement 2004). 

Theory of Action: How Conversion of Neighborhood Schoois to 
Magnet Schoois Couid improve Student Outcomes 

The key features of a magnet school are a specialized curriculum (e.g., performing arts or 
mathematics and science programs) or instructional method (e.g., open classrooms or team 
teaching), and enrollment that is open to students from outside the school neighborhood or 
attendance zone.^ The specialization in curriculum or teaching method is hypothesized to affect 
the school in two ways (Steel and Levine 1994). First, it is the vehicle to change the makeup 
of the student body, by attracting students from outside the school’s neighborhood who are 
different from students already attending the magnet school — in terms of their race/ethnicity, 
socioeconomic status, or academic achievement. Second, the specialized curriculum or teaching 
method can provide a focus for teacher professional development and collaboration, and therefore 
has the potential to improve the learning environment and instruction, in turn boosting student 
achievement for all students. 

However, the specific path by which this theory of action might result in increased student 
diversity and achievement depends on the characteristics of the neighborhood school before it 
converted. There are two models or approaches, both supported through MSAP grant funds. 
Because these two models start with schools in different situations, and the schools are expected 
to recruit different types of students and have different target populations of policy interest, it is 
important to differentiate between them. 


Rulings against race-conscious strategies culminated in 2008 guidance from ED’s Office of Civil Rights that 
“strongly encourages the use of race-neutral methods for assigning students to elementary and secondary schools.” 
(Office for Civil Rights “Dear Colleague” letter dated August 28, 2008; no longer publically available). 

^ In districts with more than one school for the same level of education (e.g., elementary schools), officials typically 
establish geographic areas around each school to define the households whose children will be assigned to or can 
automatically attend the school. 
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The Traditional Magnet Model 

One long-standing approach is for a low-performing neighborhood school that is serving students 
from low- income households or minority racial/ethnic groups to convert to a magnet school by 
adopting a specialized curriculum or instructional method (Exhibit 1.1). It is hoped that this 
change will enable the school to attract students of racial or ethnic groups different from the 
students who live in the school’s neighborhood or attendance zone, as well as students who are 
more economically advantaged or higher achieving. 

Both the new students attracted to the school from outside the neighborhood and the 
neighborhood students could benefit from the school’s specialized curriculum or instructional 
method. If the recruitment of new students is successful, the hypothesis is that there will be a 
“spillover” effect. The higher achieving students from outside the neighborhood will contribute to 
higher teacher expectations. This, combined with the new students’ presumed stronger academic 
motivation, will lead to improvements in the behavior and achievement of the neighborhood 
students (Bloom et al. 2004; Christenson et al. 2004). In addition, more economically advantaged 
students might bring with them parents who are able to more effectively advocate for ongoing 
improvements in the converted schools (Stevenson and Baker 1987). Finally, the race or ethnic 
diversity of students drawn from outside the neighborhood could support student learning by 
exposing all students in the school to a wider range of life experiences and opinions than they 
would otherwise have (see, e.g., Harris 2010). 

In this approach to magnet conversion, the main goal is to improve the outcomes of the 
neighborhood students who would ordinarily attend the low-performing school. These students 
are typically low achieving and from low-income households and are the focus of most federal 
policy concern. The newly recruited students are also expected to benefit from the match 
between their interests and the schools’ specialized curriculum and instruction, as well as from 
the improved learning environment in the magnet school. 

Exhibit 1.1. Theory of Action for Conversion of a T raditional Magnet School 



3 









What Happens When Schools Become Magnet Schools? 


The Destination Magnet Model 

Another approach is to convert a relatively high-performing school, often serving relatively 
low-minority and higher income student populations, to a “destination” magnet (Exhibit 1.2).^ 
Like the traditional magnet schools, destination magnet schools are expected to adopt a 
specialized curriculum or instructional methods. The hope is that adopting the curriculum will 
make the school an opportunity or “destination” for students from outside the neighborhood, who 
frequently attend struggling schools, are lower achieving, more economically disadvantaged, and 
more likely to be from minority racial/ethnic groups than are the neighborhood students. 

Like the process for traditional magnet schools, the adoption of a specialized curriculum or 
instructional methods is expected to improve the learning of both neighborhood students and 
students from outside the neighborhood. In addition, under the destination model, the students 
from outside the neighborhood are expected to benefit from attending school with advantaged 
peers from the neighborhood. It is the outcomes for students from outside the neighborhood that 
constitute the primary policy interest for destination schools. 

Exhibit 1 .2. Theory of Action for Conversion of a Destination Magnet School 



® Destination schools are typically relatively low-minority, economically advantaged, and/or high-performing 
schools compared with other schools in their district. They may, however, not necessarily be low minority, 
economically advantaged, or high performing compared with all schools in their states or nationally. Destination 
schools should not be confused with the competitive-admission schools mentioned earlier, such as Boston Latin, 
which are highly selective and typically do not have designated neighborhood students they are required to enroll. 
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Why This Study? 

Despite the widespread presence of different types of magnet schools in the United States and 
growing interest in their outcomes, there is limited evidence about their effectiveness based on 
studies using rigorous designs.’ Previous research has had drawbacks: 

■ Most studies have not distinguished among different types of magnet programs, which 
could obscure their outcomes or impacts. Prior research has not separated traditional from 
destination magnet conversions, or new, elementary, schoolwide conversion magnet 
schools from other magnet approaches (i.e., schoolwide programs versus individual 
programs within a school; new programs versus mature ones; magnet elementary schools 
versus magnet high schools). Because of this lack of distinction, previous studies cannot 
provide information about the outcomes of any particular approach. 

■ Some studies have combined the results for neighborhood students and students from 
outside the magnet neighborhood or focused solely on new students drawn by the magnet 
program. However, the hypotheses behind magnet conversion suggest that each group of 
students is affected differently by magnet conversion. 

■ Some studies have been limited to individual districts, providing less confidence that the 
results apply to other locations. 

■ Overall, the research has yielded mixed results. Although some studies have found 
positive effects for magnet programs, others have found negative effects or no effect. 

This study addresses some of the limitations of the previous research. It separately describes the 
outcomes for schools following the destination approach and schools following the traditional 
approach. It focuses on neighborhood elementary schools that convert to whole-school magnets. 
It separately describes what happens to the schools overall as well as the two groups of students 
that they serve (neighborhood students and students from outside the magnet neighborhood) in 
terms of diversity and achievement. Finally, this study includes 21 magnet schools in 1 1 districts 
across the United States. 

This study is, however, not without its own limitations. The most rigorous evaluation method 
would involve randomization (e.g., randomly assigning some schools to convert to magnet 
schools and others to serve as controls or randomly assigning some students to attend conversion 
magnet schools and others to attend neighborhood public schools).* But because this study 
examines conversion schools already funded by the MSAP under the standard MSAP application 
process, random assignment of schools or students was not feasible. Instead, the study used a 
statistical approach to compare changes in diversity and achievement in conversion magnet 
schools to changes in other neighborhood schools in their districts that did not convert. Although 


’ A thorough review of the literature on magnet schools and achievement can be found in Christenson et al. (2004), 
which reviews 1 1 studies of magnet schools and achievement. A review by Ballou (2009) provides more recent 
evidence from eight studies, five of which use an evaluation methodology that is considered rigorous. 

* Although random assignment of schools might be possible in principle, historically, MSAP has provided grants for 
specific schools that districts choose to convert. And although admission of students by lottery might be feasible, 
none of our study schools had more applicants than available spaces, preventing the use of lotteries. 
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findings from these statistieal models provide additional information about the relationship 
between magnet eonversion and ehanges in diversity and aehievement, they are not eonelusive 
evidenee of the effeetiveness or laek of effeetiveness of magnet sehool eonversion. 

Also, although some MSAP eonverting sehools set goals for improving diversity for speeifie 
raeial/ethnie groups (e.g., inereasing the proportion of Latino students but not Afriean- 
Amerieans), the study grouped all students from minority raeial/ethnie baekgrounds together. 
Thus, the study might have examined some ehanges in diversity that were not a foeus of the 
sehools’ efforts. 

Organization of the Report 

The remainder of this report deseribes the study methods and findings in more detail. 

■ Chapter 2 deseribes how MSAP-funded sehools and distriets were seleeted for the study 
and the analytie approaehes used to examine them. 

■ Chapter 3 puts the study’s eonversion magnet sehools into eontext, deseribing faetors that 
eould have infiueneed how sehool diversity and aehievement may have ehanged in the 
sehools as they eonverted. 

■ Chapter 4 deseribes the MSAP-funded eonversion aetivities in study magnet sehools. 

This ehapter explores the type and the extent of magnet eonversion implemented, whieh, 
like eontext, may affeet diversity and aehievement. 

■ Chapter 5 reports the results for traditional magnet sehools, ineluding whether they 
attraeted students from outside the neighborhood and improved in diversity and 
aehievement, and whether sueh ehanges in diversity and aehievement might be attributed 
to magnet eonversion rather than other faetors. Chapter 6 reports the eorresponding 
results for the destination magnet sehools. 

■ Finally, Chapter 7 diseusses variation in results aeross study sehools and suggestions for 
future work to better understand outeomes for magnet sehools. 
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Chapter 2: The Study Schools and Methods 

Examining how public schools change when they convert to become magnet sehools requires a 
group of eonverting sehools to observe. Beeause the study aimed to deseribe what happened to 
the schools’ diversity and student aehievement, and also to see whether the eonversion could 
have played a role in those outcomes, schools meeting speeifie criteria were needed, along with 
extensive data on the schools and their students. This chapter provides an overview of the criteria 
for seleeting the study sehools, a deseription of the districts and schools included, a discussion of 
the data colleeted, and an overview of the analysis methods used. Additional details are provided 
in Appendices A through F. 

Selection of Districts and Schools 

The study sought to assess the ehanges that oeeurred in a recent set of MSAP districts and 
schools. At the time the study began, the 2004 and 2007 MSAP grant eycles were the most 
reeent, and sehools were seleeted from both cyeles in order to meet the study’s target sample size 
of at least 13 schools.^ 


The study focused on magnet schools that were elementary, “schoolwide” conversions. 


The sample of districts and schools was selected to meet several criteria to facilitate the planned 
analyses. First, the study sample ineluded only elementary schools. Elementary schools were 
seleeted because they were the most eommon type of magnet school conversion among MSAP 
grantees. 

Seeond, the sample was restricted to schools that had adopted “sehoolwide” magnet programs; 
that is, magnet programs serving all students in the sehool. It exeluded programs serving only 
some students in a school. These programs-within-a-school were excluded beeause the number 
of students in a school participating in these programs was potentially too small to support 
analysis. 

Third, the schools selected were “new” magnet eonversions rather than schools that were already 
magnet sehools when they obtained MSAP support. The analysis approach involved eomparing 
sehools before and after they beeame magnet sehools, which would not have been possible with 
pre-established magnet sehools. 

Finally, the sample was limited to sehools that served neighborhoods whose student attendanee 
boundaries did not ehange during the study period. The goal of the study was to compare school 
diversity and student learning before and after conversion and to investigate whether magnet 


^ For a discussion of the approach taken to estimate the number of schools needed (also called statistical power 
calculations), see Appendix F.2.2.3.1. Although a minimum of 13 schools was needed, the study recruited as many 
schools as possible and was ultimately able to include 21 in the analysis. 
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conversion might have played a role in any ehange that was observed. If the boundaries of the 
neighborhood served by the sehool ehanged over that period, it would be diffieult to determine 
whether ehanges in outeomes were related to the eonversion or to the ehanges in the boundaries 
In the 2004 and 2007 funding eyeles, MSAP supported a total of 1 19 elementary, sehoolwide 
magnet sehool eonversions in 41 distriets. 


Twenty-one magnet schools in 11 districts were abie to provide detaiied records on 
student characteristics, achievement, and schooi assignments. 


The sample was limited to sehools in districts that could provide the data needed for the study, 
ineluding; 

■ Individual students’ raee/ethnieity, soeioeeonomie baekground (eligibility for free or 
redueed-priee luneh), and aehievement, beeause these are the key outeomes of magnet 
eonversion. 

■ Whether students lived in the sehool’ s neighborhood or outside the neighborhood, in 
order to foeus separately on eaeh student group. 

■ Consistent measures and ways of reeording students’ eharaeteristies and aehievement by 
the distriet for the years ineluded in the study, to enable the measurement of ehange over 
the period before and after eonversion. 

■ Data for students in all elementary sehools in the distriet. The analysis involved two 
steps — first eomparing eonversion magnet sehools to their distriets as a whole, and then 
eomparing them to other neighborhood sehools that did not eonvert (i.e., sehools that 
primarily served students within their neighborhood, exeluding other magnets and 
eharters). 

After reviewing information and/or speaking with representatives of the 41 potentially eligible 
distriets, the study found that 21 eonversion magnet sehools in 1 1 distriets among the MSAP 
grantees met the study’s stable neighborhood boundary and data requirements. All 21 sehools 
and their distriets were ineluded in the study sample (See Appendix A for more on sample 
seleetion).'' In eaeh of these 1 1 distriets, there were only one or two eonverting magnet sehools 

In this study, “economically disadvantaged students” are defined as students who are eligible for a free or 
reduced-price lunch. Students are eligible if their family income is below 185 percent of the poverty line. 
Information on eligibility was not available for three districts (six schools) that were, nevertheless, included in the 
sample. For one district (two schools), the lack of information was known prior to the decision to include it in the 
study. As planned, this district was not included in the analysis of economic disadvantage, but information on parent 
education was available and was used as a substitute for free or reduced-price lunch as a control variable in the 
analysis of achievement. For two districts (four schools), information on free and reduced-price lunch was provided 
but later found to be unreliable. See Appendix F. 4.2.1 for more information. 

" All districts and schools that met the study criteria and provided the required data were included in the study. 
While 22 study magnet schools were selected for the study, two were located in the same building and had been one 
school prior to conversion. These two co-located schools were treated as one school in the analysis. Three schools in 
two districts were selected from the 2004 funding cycle, and 19 schools in 10 districts were selected from the 2007 
funding cycle. One district contained two study magnet schools selected from the 2004 funding cycle and two other 
study magnet schools selected from the 2007 funding cycle. 
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included in the study, but at least 10 other neighborhood elementary sehools that did not eonvert. 
(See Exhibit 2.1, which lists the distriets included in the study, using anonymous codes to 
distinguish them.) 


Exhibit 2.1 . MSAP Grant Years and Numbers of Public Elementary Schools 
Served by Study Districts 


District 

MSAP Grant Year 

Study Magnet Schoois 

Neighborhood Pubiic 
Schoois 

A 

2004 

1 

169 

B (2004) 

2004 

2 

14 

B (2007) 

2007 

2 

14 

C 

2007 

1 

15 

D 

2007 

3 

78 

E 

2007 

1 

163 

F 

2007 

3 

15 

G 

2007 

2 

74 

H 

2007 

1 

22 

I 

2007 

1 

65 

J 

2007 

2 

91 

K 

2007 

2 

18 

Total 


21 

724^ 


t Numbers of neighborhood public schools from District B were counted only once in this total. 

NOTE: Two conversion magnet schools located in the same building that had been one regular neighborhood school prior to 
conversion are counted as one school in this table. 

SOURCE: District administrative data. 


Study districts, iike MSAP districts overaii, were iarger, were more urban, and served a 
higher share of minority students than districts nationwide. 


Although the districts in the study were not ehosen to be representative of all districts that 
receive MSAP funding or of all districts in the United States, it is important to understand how 
they are different or similar. These eomparisons provide some boundaries for using the study 
findings in a larger eontext, even though the results eannot be generalized to the MSAP program 
as a whole. 


All comparisons were made in the year before conversion: in the 2003-04 school year for the 2004 cohort and in 
the 2006-07 school year for the 2007 cohort. 
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■ Study districts were similar in size to MSAP distriets (100 pereent and 78 percent, 
respeetively, with more than 10,000 students) but larger than U.S. publie sehool districts 
overall (6 pereent with more than 10,000). 

■ Study distriets, like all MSAP distriets, were mostly urban (75 and 70 pereent 
respeetively), whereas only a small pereentage of distriets in the United States were urban 
(12 pereent; Exhibit 2.2). 

■ The proportion of students from minority raeiahethnie baekgrounds in study districts was 
similar to all MSAP-funded districts (63 and 67 percent respeetively) and both were 
signifieantly higher than the proportion in all U.S. public school districts (27 percent; 
Exhibit 2.2). 

■ The proportion of students from eoonomieally disadvantaged baekgrounds in study 
distriets (49 pereent) was also similar to all MSAP-funded distriets (55 pereent) and all 
U.S. publie sehool distriets (40 pereent); however, the proportion in all MSAP-funded 
distriets was signifieantly higher than in all U.S. publie school districts (Exhibit 2.2) 

■ The 12 distriets ineluded in the study, as well as MSAP sehools, represented all four 
U.S. regions. 


One study district containing magnet schools from both funding cycles is represented twice in background 
exhibits to display the differing context for 2004 and 2007 funding cycle magnet schools in this district. Hence, the 
count of districts here, and in other exhibits when noted, is 12 rather than 11. 
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Exhibit 2.2. Proportion Urban and Average Proportion of Minority and Disadvantaged 
Students Served by Study Districts, All MSAP Districts, and All U.S. Public 
School Districts 
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NOTE: N = 12 for sample districts with District B represented twice in the data, once for the 2004 and once for the 2007 MSAP 
funding cycles. For all MSAP and all districts nationally, percentages in the exhibit are a weighted average of the 2003-04 and 
2006-07 percentages, where the 2003-04 percentages were weighted 2/12 for the two 2004 funding cycle study districts 
represented in the exhibit, and the 2006-07 percentages were weighted 10/12 for the ten 2007 funding cycle districts represented in 
the exhibit. * All study districts significantly different from all U.S. public school districts (p < .05); + All study districts significantly 
different from all MSAP districts (p < .05); f All MSAP districts significantly different from all U.S. public school districts (p < .05). 

SOURCE; Common Core of Data 2003-04 and 2006-07. 

Classification of the Study Schools as Traditional or Destination 
Magnet Schools 

As described in Chapter 1, the theory of action is somewhat different for traditional and 
destination schools. Thus, the analysis was conducted separately for schools of each type. Data 
on achievement levels and student characteristics before conversion were used to classify each of 
the 21 study schools as either a traditional or destination magnet school, though schools did not 
always fit neatly into the categories. 

■ A school was classified as a traditional magnet school if, prior to conversion, its average 
achievement in ELA and in mathematics were both below the district average, and the 
proportion of students in the school from minority backgrounds and the proportion from 
disadvantaged backgrounds were both above the proportion for the district as a whole. 
Fourteen schools met all four of these criteria. Three additional schools that met three of 
the four criteria were also classified as traditional. 


The percentage of students from economically disadvantaged backgrounds was not available for six schools. On 
the basis of their other characteristics, all six were classified as traditional magnet schools. 
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■ A school was classified as a destination magnet school if, prior to conversion, its average 
aehievement in ELA and in mathematies were both above the distriet average, and the 
proportion of students in the sehool from minority baekgrounds and the proportion from 
disadvantaged baekgrounds were both below the district proportion. No school met all of 
these eriteria; however, four sehools that met three of the four eriteria were elassified as 
destination. 

Using this approaeh, 17 of the 21 eonversion magnets ineluded in the study were elassified as 
traditional magnet sehools, and 4 were classified as destination magnet sehools (see Appendix 
A. 5 for more details). 

Data Collected for the Study 

To conduct the analysis (discussed in detail below), the study relied primarily on student 
administrative reeords eolleeted by the district for state reporting purposes and obtained directly 
from district offices. These reeords eontained data on student eharaeteristics, aeademie 
aehievement based on state assessments, and neighborhood of residenee for all elementary 
students in each district. Typically, the data obtained ineluded all elementary students enrolled in 
eaeh distriet during the two years before conversion and the four years after eonversion.'^ 
However, beeause of differenees in when state assessments were offered, the number of years 
and grades used in the study’s analyses varied by distriet (Exhibit A.3).'^ Although the 
availability of data on students’ status as a neighborhood or out-of-neighborhood student was a 
eriterion for inelusion of districts and schools in the study, for 6.0 pereent of the student reeords 
in the sample, resideney information was missing. In 2.0 pereent, it was possible to impute the 
information using data for that same student from other years, leaving 4.0 pereent of reeords in 
the analysis dataset missing resideney information. The final dataset used in the analysis 
eontained one reeord for eaeh student for eaeh year during the study period that the student 
attended a sehool in the distriet — about 1,500,000 student reeords in all (see Appendix A.3 for 
more detail). 

The study also gathered information from other sourees to put the magnet eonversions in eontext 
and deseribe magnet implementation. 


The official MSAP grant period is three years: 2004-05 through 2006-07 for the 2004 funding cycle and 2007-08 
through 2009-10 for the 2007 funding cycle. Because of delays in awarding the 2007 MSAP grants, the districts in 
the study sample were permitted to continue implementation activities using remaining grant funds, and the study 
continued to collect student data, through the 2010-1 1 school year. The study also collected data from districts in 
the 2004 MSAP funding cycle for one or more years after the end of their three-year grant period. Accordingly, the 
study was able to track composition and achievement outcomes for four (rather than three) years after conversion in 
all but one district. The exception was for District B in the 2007 funding cycle: changes in the administrative data 
collected by the district prevented the analysis from being extended to the fourth post-conversion year. 

For details on state assessments see Appendix A. 

For details on determining students’ neighborhood of residence and the method used to handle missing 
neighborhood information, see Appendix B.2. 
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■ Systematic interviews were conducted with MSAP directors in 201 1 in all 1 1 districts; 
the directors were asked for retrospective information on district improvement policies 
and on study magnet schools, including information on difficulties schools might have 
had in student recruitment and retention (see Appendix C for details). Magnet school 
principals were surveyed to obtain information on the conversion magnet schools, 
including magnet conversion components and implementation timelines (responses were 
received from all but one principal; see Appendix D for details). 

■ Information on student characteristics and other characteristics of schools and districts 
was collected from additional existing data sources, including those maintained by ED 
(school-level data from the EDFacts system based on state annual performance reporting 
and the National Center for Education Statistics’ CCD), as well as state education 
department websites (see Appendix E for details). 

Examining Change in Magnet Schoois 

This study addresses three questions to see if conversion schools changed in ways that would be 
expected based on the theory of action; 

1 . Did the composition of neighborhood students and students from outside the 
neighborhood in the magnet schools change after conversion? 

2. Did the diversity in the magnet schools increase after conversion, and is there any 
evidence that it was related to conversion? 

3. Did the achievement in the magnet schools increase after conversion, and is there any 
evidence that it was related to conversion? 

1. Did the composition of neighborhood students and students from outside the 
neighborhood in the magnet schools change after conversion? 

Both traditional and destination magnet schools were expected to bring in more students from 
outside the neighborhood after conversion, and for the outside students to be different from the 
neighborhood students on various characteristics. To assess whether the first happened, the 
proportion of students from outside the neighborhood in the study schools before conversion was 
compared to the proportion afterwards. Then the characteristics of students from outside the 
neighborhood were compared to the characteristics of neighborhood students — the proportion 
from racial/ethnic minority groups, the proportion who were economically disadvantaged 
(i.e., eligible for the federal free and reduced-price lunch program), and the average standardized 
test scores after conversion. Einally, because there were students from outside the neighborhood 


Principal surveys were administered in summer and fall 2008 for 2004 funding cycle schools and fall 2010 
through summer 2012 for 2007 funding cycle schools. Two conversion magnet schools that operated at the same 
location and had been one school during early years of the study time period were counted as one school for most 
analysis in the report. However, as the schools had different principals at the time of survey administration, each 
principal was administered a survey, bringing the total number of surveys administered to 22. Responses were 
collected from 21 of 22 study magnet school principals (95 percent). 
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already in the magnet sehools prior to eonversion, an additional step examined whether the 
differenees between the students from outside the neighborhood and the neighborhood students 
were greater after eonversion. 

2. Did the diversity in the magnet schools increase after conversion, and is there 
any evidence that it was related to conversion? 

This question foeused on the diversity in magnet sehools in terms of the proportion of minority 
students and students from disadvantaged baekgrounds. The analysis proeeeded in two steps. 

Describing changes in magnet schools and their districts. As a first step, the pre- to 
postconversion change in magnet schools was compared with the change in the districts in which 
the magnet schools were located. The goal of this analysis was to determine whether magnet 
schools changed in the anticipated direction relative to any overall change in their districts, and if 
so, how large the changes were. 

The diversity in a magnet school was defined as the proportion of students from minority 
racial/ethnic groups and the proportion who were economically disadvantaged served by the 
magnet schools, relative to the same proportions served by their districts overall. This “relative” 
approach was used for several reasons. First, the characteristics of a converting school could 
change simply because the demographics of the district are changing; anchoring the magnet 
school change in the district change helps to take this possibility into account. Second, there is 
typically a set pool of students in a given district. Thus, when a converting magnet draws a 
greater or lesser share of some student groups, it invariably means a change in some other school 
or schools in the district. That outcome is desirable if the changes result in greater balance in 
student composition across the district. The MSAP gives districts and schools flexibility in how 
they measure improvements in diversity. For this study, an improvement in diversity is defined 
as an increase or decrease in the “concentration” of students from certain groups defined by 
race/ethnicity or socio-economic status in magnet schools in ways that would be expected based 
on the theory of action. As a result, in this study a positive change in diversity means that the 
proportion of students who are disadvantaged or from a minority race/ethnic background in the 
converting magnet schools became more like that of their districts. 

Investigating the role of magnet conversion. Once the overall change from pre- to 
postconversion was examined, a second analysis was conducted to investigate whether magnet 
conversion may have played a role in any changes in diversity in magnet schools. In an effort to 
rule out other potential factors that might explain the observed change, the changes in magnet 
school student composition were compared with the changes in neighborhood elementary 
schools that did not convert (i.e., schools that primarily served students within their 
neighborhood, excluding other magnets and charters). In this analysis, the average change in 
neighborhood schools that did not convert represents a “counterfactual” — in other words, 
changes in neighborhood schools that did not convert represent what would have occurred in 
magnet schools if they had not converted. If the magnet schools experienced changes similar to 
what would be predicted based on the changes experienced by neighborhood schools that did not 
convert, it would suggest that the magnet school outcomes might be part of a larger trend for all 
neighborhood schools in the district rather than a result of conversion. If the changes were 
different than what would be predicted based on neighborhood schools that did not convert. 
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then magnet eonversion is a possible explanation for the differenee. (The statistical method, a 
“comparative interrupted time series,” or CITS, analysis, is described in Appendix F.2.2.2.) 
Although it is not possible to rule out all factors other than conversion with certainty, given the 
study design it is possible to provide suggestive evidence of the role of conversion. 

Because some schools had a higher proportion of minority students or disadvantaged students 
than their districts and others had a lower proportion, the key variable in this analysis was the 
size (or absolute value) of the difference between the school and district proportions. Regardless 
of proportion prior to conversion, improved diversity entailed a reduction in the difference 
between the two proportions. Hence, the analysis used the size of the difference between a 
school and its district’s proportions to examine whether magnet schools became more like their 
districts after conversion than would be predicted. 

3. Did the achievement in the magnet schools increase after conversion, and is 
there any evidence that it was related to conversion? 

A primary goal of magnet conversion is to improve the achievement of all students in the school. 
Paralleling the approach for research question 2, this research question was addressed in two 
steps. 

Describing changes in magnet schools and their districts. As a first step, the study examined 
whether the average mathematics and English language arts (EL A) achievement in the magnet 
schools was higher after conversion, by comparing the test scores of students attending after 
conversion to the test scores of students who were in those same grades in the years before 
conversion. Eor comparison, a similar analysis was conducted of achievement levels in the 
districts in which the magnet schools were located. Higher achievement in magnet schools after 
conversion could be due to either improved learning among students who attended the schools, 
or to the attraction of higher performing students after conversion who, essentially, bring up 
average test scores. Both could be a result of the conversion to a magnet school, or of other 
factors such as district-wide initiatives. 

Investigating the role of magnet conversion. As a second step, an analysis was conducted to 
examine whether conversion played a role in the observed changes in achievement. To assess 
the potential role of magnet conversion, the year-to-year achievement gains of students in 
neighborhood schools that did not convert were used to predict what the gains for neighborhood 
students in study schools would have been had the study schools not converted. (The statistical 
method, a comparative interrupted time series, is described in Appendix E.2.2.2.) To focus the 
analysis on what students learned while attending a magnet school rather than on what they had 
learned before, the analysis focused on students’ annual achievement gains — their increases (or 
decreases) from one grade to the next. In addition, the statistical model also accounted for the 
race/ethnicity, economic status, age, disability status, and English language learner (ELL) status 
of students so that the gains for neighborhood students in study schools were compared with 
students with similar background characteristics in neighborhood schools that did not convert. 

The analysis focused only on the traditional magnets. The neighborhood student populations in 
traditional magnet schools — typically higher minority, more economically disadvantaged, and 
lower achieving — are of the greatest policy interest. In destination schools, the policy interest is 
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the lower performing students from outside the neighborhood, and there are better approaehes to 
examine the outeomes of students who transfer into sehools, ineluding methods based on 
lotteries. However, there were too few students from outside of the neighborhood transferring to 
destination magnet sehools to apply these methods reliably. 
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Chapter 3: Magnet School Context 

The settings in which magnet schools are located could shape the changes that occur in these 
schools after conversion. Certain district characteristics or efforts could influence the magnet 
schools’ ability to attract and retain students from outside the neighborhood or to distinguish 
themselves from other district schools. This chapter explores district factors that could, 
hypothetically, affect study magnet schools’ success. 


Most study districts had diverse popuiations from which conversion magnet schoois 
couid draw. 


The overall composition of students in the districts could influence the magnet schools’ ability to 
draw students from outside their neighborhoods who are different from students already 
attending these schools. Depending on whether a magnet implements the traditional or the 
destination approach, the types of students targeted for recruitment likely differ. If the type of 
student that a magnet targets is a small share of the district’s student population, the school might 
experience difficulty bringing in enough students to change the magnet school’s student body. 
For example, schools implementing the traditional approach might have a hard time attracting 
economically advantaged students if their districts largely serve economically disadvantaged 
students. The converse is true for the destination schools. Districts with fewer than 20 percent or 
more than 80 percent of their students with the characteristics of interest were considered less 
diverse in ways that could present challenges for the success of magnet recruitment, whichever 
type of magnet approach was pursued. 

However, most of the magnet schools in the study did not appear to face those conditions (Exhibit 
3.1). 

■ More than half of the study districts had a mix of white students and students from 
minority racial/ethnic groups. In 7 of the 12 study districts, representing 12 magnet 
schools, the population of students in minority racial/ethnic groups was between 

20 percent and 80 percent. Of the six study schools in the four districts with more than 
80 percent of the students from minority raciahethnic groups, three were traditional 
magnet schools that could potentially be constrained by the more limited student 
diversity and three were destination magnet schools. There was one destination magnet 
in the one district with relatively few students from minority racial/ethnic groups (less 
than 20 percent of the district population). 

■ All but one district, with one destination magnet school, served an economically 
diverse student body. In 1 1 of the 12 study districts, representing 18 magnet schools, the 
population of students who were economically disadvantaged was between 20 percent 


The sensitivity of the changes in outcomes after conversion to these potential constraints in the pool of targeted 
students was tested. See Appendix H. 1 . 
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and 80 percent. There was one destination magnet in the one district with relatively few 
economically disadvantaged students. 


Exhibit 3.1 . Number of Study Districts by Range of Percentage Minority Students 
and Percentage Disadvantaged in the Year Before Magnet Conversion 
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NOTE: A/ = 12 districts for this exhibit. District B is represented twice to display differing context for the 2004 and 2007 funding cycle 
magnet schools in this district. 

SOURCE: Common Core of Data 2003-04 and 2006-07 school years. 


In study districts, the magnet schools were not the only choice option available to 
parents and students, and not necessarily the better performing of the options. 


The availability of other public school choice options (i.e., other magnet and charter schools) and 
their relative academic performance also could influence study magnet schools’ ability to recruit 
and retain students. More options could mean more competition for students. And to the extent 
that parents seek out and make their school choices based on test scores, the converting magnet 
schools could be at an advantage or a disadvantage relative to these other choice options or to the 
traditional public schools that students could attend in their home neighborhoods. 

■ Study magnet schools were located in school districts with many school choice 
options. In study districts, on average, 22 percent of the elementary schools were charter 
or magnet schools (Exhibit 3.2). This was approximately three times the national 
percentage (7 percent of schools). The percentage of elementary schools that were charter 
or magnet schools was lower than the national percentage in only one district (District E), 
where 1 percent of the schools were charter or magnet schools. 

■ The other choices available to families were not uniformly better or worse 
academically than the converting magnet schools, as measured by their program 
improvement (PI) status. PI status, a designation that is part of the federal school 
accountability system and made public to parents, is given to schools that failed to meet 
adequately yearly progress (AYP) in achievement in a given subject or in the percentage 
of students tested for two consecutive years. In four study districts (A, B 2004, J, and K), 
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seven sehools that eonverted to magnet schools had not been in improvement status, 
whereas other choice and traditional public schools had received that designation 
(Exhibit 3.3). In the other eight districts, a similar or higher share of the magnet 
conversions than other schools had been in PI status. 

Exhibit 3.2. Percentage of Elementary Public Schools in Study Districts 
and the United States That Were Charter or Magnet Schools 
in the First Year of the MSAP Grant 
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NOTE: District B is represented twice to display differing context for the 2004 and 2007 funding cycle magnet schools in this district. 
CCD identifiers of magnet and charter schools were verified and, in some instances, corrected with data from the MSAP office, 
district websites, and MSAP director interviews. 

SOURCE: CCD 2004-05 and CCD 2007-08. 


Exhibit 3.3. Number of Study Magnet Schools, Percentage of Other Magnet 

and Charter Schools, and All Other Schools Ever in Program Improvement 
(PI) During the Postconversion Period 


District 

Study Magnet Schoois 

Other Magnet and 

Aii Other Schoois 


N 

Ever in Pi 

Charter Schoois 


A 

1 

0% 

42% 

54% 

B (2004) 

2 

0% 

33% 

29% 

B (2007) 

2 

50% 

29% 

30% 

C 

1 

100% 

20% 

62% 

D 

3 

100% 

76% 

62% 

E 

1 

100% 

50% 

59% 
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District 

Study Magnet Schoois 

Other Magnet and 
Charter Schoois 

Aii Other Schoois 

F 

3 

100% 

N/A 

13% 

G 

3 

33% 

37% 

20% 

H 

1 

100% 

75% 

45% 

1 

1 

100% 

71% 

80% 

J 

2 

0% 

42% 

32% 

K 

2 

0% 

50% 

20% 

Combined 


55% 

52% 

50% 


NOTE: District B is represented twice to dispiay differing context for the 2004 and 2007 funding cycle magnet schools in this district. 
In District F, there were no other magnet or charter schools. 

SOURCE: EDFacfs and state accountability websites. 


Study schools were converting to magnet schools during a period in which their districts 
were improving academicaiiy and implementing a variety of reforms. 


The introduction of district activities designed to improve achievement, whether targeted to 
particular schools or implemented districtwide, has the potential to affect the converting magnet 
schools. Education reforms taking place in nonmagnet schools could make it more difficult for 
newly created magnet schools to distinguish themselves academically. District reforms the 
magnet schools had to undertake (because they were required for all schools in the district or just 
the low-performing schools) could interfere with the simultaneous implementation of a new 
schoolwide magnet program. 

■ Study schools were located in districts that were, on average, higher achieving in the 
postconversion period relative to the preconversion period. Study districts 
experienced an average increase of 6 percentile points in ELA and 8 percentile points in 
mathematics (Exhibits 3.4 and 3.5, respectively). Eive districts (containing nine study 
schools) experienced an increase in ELA achievement, whereas seven districts 
(containing 14 study schools) experienced an increase in mathematics achievement. No 
district experienced a significant decrease in achievement in either subject. 

■ Almost all the study districts required schools to undertake reform activities during 
the time that the conversions were being implemented, according to MSAP 
directors. In seven (of the eight) reporting districts, MSAP directors reported that at least 
one reform activity was required of all elementary schools in the district (Exhibit 3.6).^^ 
In four of these districts, at least one additional reform activity was required for schools 
in PI status. In the districts requiring reform activities in PI schools, all of the conversion 
magnet schools in the district were in PI status. 


District B is represented twice, once for each funding cycle. 

Information on district reforni activities was collected retrospectively as part of MSAP program director 
interviews in which directors were asked about reforms that were implemented in their districts. This information 
was not collected for the 2004 funding cycle for schools in Districts A and B. 
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Exhibit 3.4. Average ELA Achievement Percentile for Districts, Pre- and Post- 
conversion 



■ Preconversion ■ Postconversion 


* The difference between pre- and postconversion achievement is statisticaiiy significant (p < .05). 

NOTE: District B is represented twice to dispiay the differing contexts for the 2004 and 2007 funding cycie magnet schoois in this 
district. District data were standardized within grade and district using the first year of data as a base year. Statisticai tests were 
based on annuai achievement averages for each district; average achievement was compared for the years before and after 
conversion. 

SOURCE: District administrative data. 
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Exhibit 3.5. Average Mathematics Achievement Percentile for Districts, Pre- and 
Postconversion 



0 10 20 30 40 50 60 70 80 


■ Preconversion ■ Postconversion 


* The difference between pre- and postconversion achievement is statisticaiiy significant (p < .05). 

NOTE: District B is represented twice to dispiay the differing contexts for the 2004 and 2007 funding cycie magnet schoois in this 
district. District data were standardized within grade and district using the first year of data as a base year. Statisticai tests were 
based on annuai achievement averages for each district; average achievement was compared for the years before and after 
conversion. 

SOURCE: District administrative data. 


22 


What Happens When Schools Become Magnet Schools? 


Exhibit 3.6. Districtwide School Reform Activities Required in Years After Conversion for All Schools, Schools in PI 
(Only PI), or None of the Schools for Study Districts With the 2007 Funding Cycle Study Magnet Schools 


District 

Study Magnet 
Schoois (A/) 

Adopt New 
Mathematics 
Curricuium 

Adopt New 
ELA 

Curricuium 

increase 

instruction 

Time 

Use Frequent 
Assessments 

Assign 

instructionai 

Speciaiist 

Test 

Preparation 

Activities 

B (2007) 

2 

All 

None 

All 

All 

None 

All 

C 

1 

None 

All 

All 

All 

None 

All 

D 

3 

All 

None 

All 

All 

None 

Only PI 

E 

1 

Only PI 

Only PI 

Only PI 

All 

All 

Only PI 

F 

3 

All 

None 

None 

None 

All 

Only PI 

G 

3 

None 

None 

None 

None 

None 

None 

H 

1 

All 

None 

Only PI 

Only PI 

All 

None 

1 

1 

— 

— 

— 

— 

— 

— 

J 

2 

None 

None 

None 

All 

None 

None 

K 

2 

— 

— 

— 

— 

— 

— 


NOTE: A/ = 10 study districts that contained the 2007 funding cycie study magnet schoois. Because this information was coiiected retrospectiveiy in 201 1 and 2012, it was not coiiected 
for the 2004 funding cycie schools. 

All = Activity was required for all schools in the district. 

None = Activity was not required for any school in the district. 

Only PI = Study magnet schools were in PI status, and activity was required for all district schools in PI status. 

Dash ( — ) indicates that the MSAP director did not have information to answer the question. 

SOURCE: MSAP director interviews. 


23 



This page has been left blank for double-sided copying. 



What Happens When Schools Become Magnet Schools? 


Chapter 4: Conversion Magnet School Implementation 

In addition to the district context, the specific characteristics of the school and its magnet 
program are important factors that could influence outcomes. The approach to magnet 
conversion, magnet theme, the timeline, the extent of program implementation, and the 
availability of additional resources could influence the changes in student achievement and 
composition that magnet schools are able to attain. This chapter explores those school factors 
that could, hypothetically, affect the study magnet schools’ success. 


The study magnet schools adopted a variety of themes. 


Magnet schools implement distinctive curricula to support instruction and attract students from 
outside the neighborhood or attendance zone. The MSAP-supported districts and schools 
selected the themes for their schools. Themes that are better matched to the interests of district 
students could make recruitment easier or appeal to certain kinds of students the schools want to 
serve. For example, a mathematics theme may be more appealing to students who are already 
higher achieving in this subject or students who are motivated to excel in this area. 

There was no dominant theme adopted by the conversion magnet schools in the study, although 
some fell into similar categories (Exhibit 4.1). Curricular themes related to the arts were the most 
common grouping (five schools). Four schools adopted themes related to STEM or International 
Baccalaureate, whereas three schools chose themes related to language or communication. The 
largest category in Exhibit 4.1 (other) included subject themes that did not fall into the other 
categories, such as museum studies and international studies, as well as some themes focused on 
a method of learning, such as experiential learning. 
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Exhibit 4.1 . Number of Study Magnet Schools by Magnet School Theme Category 



* Themes in the other category include both subject themes that do not faii in any of the categories iisted as weii as schooi themes 
focused around a method of iearning (e.g., experientiai iearning) rather than a particuiar subject matter. 

NOTE: Twenty-two conversion magnet schoois are represented. The two magnet schoois treated as the same schooi for anaiysis 
eisewhere are considered separateiy here as they had different themes. 

SOURCE: MSAP director interviews. 


Principals reported that almost all the study magnet programs were fully implemented 
and reached all students by the third grant year. 


Magnet school outcomes could depend on the timely and full implementation of the magnet 
program. However, magnet conversions can take some time to implement. Teachers need to be 
trained in the new curriculum theme and approach, and new students need to be recruited (U.S. 
Department of Education 2003). 

■ Principals reported that the magnet components that they considered important 
were in place hy the third grant year. Almost all the principals (more than 90 percent) 
reported mostly or fully implementing magnet program components that they viewed as 
important by the third grant year (Exhibit 4.2). However, fewer principals reported 
implementing these components in the first grant year. Eor example, approximately half 
of the principals reported mostly or fully delivering curriculum or developing or applying 
new assessments in the first grant year (53 percent and 42 percent, respectively). In 
contrast, by the third grant year, almost all the principals reported mostly or fully 
implementing these same components (95 percent). 

■ In all hut one conversion magnet school, the principals reported that all students 
were receiving instruction in the magnet curriculum hy the third grant year. 

Principals reported that in all but one conversion magnet school, all students were 
receiving instruction in the magnet curriculum by the third grant year. In the remaining 
school, more than half of the kindergarten and first-grade students were receiving 
instruction in the curriculum, but fewer than 25 percent of the students in grade 2 and 
higher were receiving instruction in the curriculum. 
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Exhibit 4.2. Percentage of Principals Reporting That Magnet Program Components 
Considered Important Had Been Implemented by the End of the First 
and Third Year After Conversion 


Providing professional development 
(out of 18 schools) 


Delivering curriculum 
(out of 19 schools) 


Hiring special staff 
(out of 13 schools) 


Developing new assessments 
(out of 19 schools) 


Applying new assessments 
(out of 19 schools) 



0% 20% 40% 60% 80% 100% 


□ Implemented by End of First Year ■ Implemented by End of Third Year 


NOTE: Nineteen principals provided information on the importance and the implementation of components. The results shown for 
each component are based on the principals who responded that the component was of major or moderate importance to the 
success of the magnet program. The number of principals reporting major or moderate importance is noted in parentheses beneath 
the component label. Percentages in the display are calculated from the number in parentheses. 

SOURCE: MSAP principal survey. 


Most study magnet schools reported that they did not experience probiems with staff 
deveiopment or student recruitment. 


Given the importance to magnet school conversion of introducing a new curricular theme and 
attracting students from outside the neighborhood, challenges in either of these areas could have 
some bearing on student and school outcomes. 
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■ Most principals reported that it was easy to build a staff that supported the magnet 
program. More than two thirds of principals (69 percent) reported that it was fairly easy 
or very easy to do so (Exhibit 4.3). 

■ MSAP directors reported that most study magnet schools did not have problems 
recruiting or retaining students because of choice in their districts or their PI status. 

According to the directors, less than one third of the conversion magnet schools 
experienced these challenges. Most MSAP directors reported that schools did not have 
difficulty recruiting or retaining students because of competition from other magnet or 
charter schools or because of conversion schools’ PI status (Exhibit 4.4). 

Exhibit 4.3. Percentage of Study Magnet School Principals Who Reported That It Was 
Very Easy, Fairly Easy, Fairly Difficult, and Very Difficult to Build Staffs 
That Actively Supported Their Magnet Programs 



Percentage 


NOTE: Percentages are based on 19 respondents to this question. 
SOURCE: MSAP principai survey. 
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Exhibit 4.4. Percentage of MSAP Directors Reporting That Other Magnet and Charter 
Schools or Program Improvement Status of Study Magnet School Posed 
Difficulty for Recruiting or Retaining Students 


Other magnet and charter schools 


Program improvement status 




Difficulty Recruiting 
■ Difficulty Retaining 


0% 20% 40% 60% 80% 100% 

Percentage 


NOTE: Percentages for other magnet and charter schools are based on 20 schools for which MSAP directors provided responses. 
Percentages for PI status are based on 22 schools. The two magnet schools treated as the same school for analysis elsewhere 
were considered separately here because they had been treated separately In the Interview with their MSAP director. All schools 
cited as having difficulty as a result of PI status also had difficulty as a result of other magnet and charter schools. 

SOURCE: MSAP director Interviews. 


More than half of the conversion magnet schools received financial support for the 
magnet program beyond the MSAP grant. 


More funding could help magnet schools implement their program or recruit students more fully 
and effectively. Of the 22 study schools, 13 received additional financial assistance from sources 
other than MSAP. This support took a number of forms, including supplemental funds from the 
district (one school); federal or state grants (five schools); financial or in-kind support from magnet 
program partners (six schools); and local grants, foundations, or fundraisers (12 schools). 


All but one study magnet school had plans to continue the magnet program after the 
MSAP grant ended. 


Continuation as a magnet school beyond the MSAP grant period may be a signal that district and 
school staff members were invested in the magnet program and viewed the magnet conversions 
as successful or developing in the right direction. MSAP directors reported that, with one 
exception, all conversion magnet schools planned to continue their programs after the grant 
period. 
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Chapter 5: Outcomes for Traditional Conversion 
Magnet Schools 

Descriptive information provided in the previous chapters indicate that the study schools felt able 
to implement their magnet programs, despite other activities and school choice options in their 
districts that could have presented challenges to them. The key issues are (1) whether the 
converting magnet schools attained the improvements in diversity and achievement that motivate 
the MSAP program and (2) whether their outcomes changed after conversion in ways that 
mirrored or differed from what would be predicted had they not converted based on other 
neighborhood schools in their districts. The latter supports the hypothesis that the conversion 
played a role in those changing outcomes, although the study design does not allow conclusions 
about the impact of the conversion to be drawn. 

Because schools engaged in the two conversion approaches described earlier — traditional and 
destination — start off with different student bodies and seek to target their recruitment to 
different types of students, it is important to assess their outcomes separately. Traditional magnet 
schools typically begin as low-performing schools serving higher proportions of students from 
low-income households or minority racial/ethnic groups. Implementing a distinctive and 
attractive academic program is expected to help them recruit students who are higher achieving, 
more economically advantaged, or more likely to help the school achieve racial/ethnic diversity. 
Both the academic program itself and the new mix of students are hypothesized to improve 
achievement. This chapter examines these outcomes for the 17 study schools that fall into this 
category of magnet school conversion. 


The traditional magnet schools served a somewhat larger share of students from outside 
their neighborhood after conversion, but these students were similar to the 
neighborhood students on a variety of characteristics. 


According to theory, one important step all converting magnet schools take is to attract students 
from outside the neighborhood who are different from the neighborhood students assigned to the 
school in terms of race/ethnicity, income, or achievement. The study schools that became 
traditional magnet schools did not appear to achieve all of these objectives. 

■ Even before the schools converted, about one fifth of their enrollment came from 
students who lived outside the neighborhood (Exhibit 5,1), On average, 21.0 percent 
of the students attending these schools were from outside the neighborhood prior to the 
conversion. This seemingly high proportion may result from preexisting choice policies 
in the districts that, as noted in Chapter 3, offered a larger share of school options for 
families than did districts nationally. Neighborhood schools in the study districts that 
did not convert to magnet schools, on average, served a similar share of students from 
outside their attendance boundaries (20.1 percent). 


Although the data in Exhibit 3.2 correspond to the first year after conversion, they are likely a good indicator of 
the availability of choice schools prior to conversion. 
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■ For the converting schools, the share of students who were from outside the 

neighborhood rose hy 5,8 percentage points (Exhibit 5,1), Across the study schools, 
the proportion of students from outside the neighborhood inereased from 21.0 percent 
before conversion to 26.8 percent afterward. The average number of students from 
outside the neighborhood increased from 108 to 137 — an additional 29 students per 
converting magnet school, which equates to about one additional student in eaeh class. 


Exhibit 5.1. Share of Students From Outside the Neighborhood in Traditional Magnet 
Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of students from outside the neighborhood In the magnet 
schools was 21 percent. This proportion Increased by 5.8 percentage points to 26.8 percent after (post-) conversion. 

NOTE: N = 17 schools In 10 districts. See Appendix F for methods used for calculations and significance testing. See appendix 
Exhibit 1.1 for additional detail. 

SOURCE: District administrative data. 

■ After conversion, the share of students from inside and outside the neighborhood 
who were from minority racial/ethnic groups or economically disadvantaged was 
similar (Exhibit 5,2), The theory of aetion predicts that successful traditional magnet 
sehools would try to attract outside students who were more economieally advantaged or 
more likely to help the school achieve racial/ethnic diversity. After eonversion, about 


Because the analysis of the share of students from outside the neighborhood was conducted using data on students 
in grades in which achievement tests were available (generally grades 2 or 3 through 5), some extrapolation was 
necessary to estimate the number of students from outside the neighborhood for all grades in each magnet school. 
First, the average attendance per grade was calculated based on the available data (85.4 students per grade). This 
number was multiplied by 6 to estimate the average attendance in grades K-5 (512 students per school), which was 
the typical grade range in study schools. The percentages in Exhibit 5.1 were then used to estimate the number of 
students from outside the neighborhood (108 students before conversion and 137 after). Because information on the 
number of classes in each grade and school was not available, it was assumed that the average class size was 
20 students (Digest of Education Statistics 2013). Based on this assumption, the increase in students from outside 
the neighborhood per class was estimated to be approximately 1 student per class (29 extra students over 
approximately 26 classes). 
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84 percent of the students from both outside the magnet neighborhood and inside the 
magnet neighborhood were members of racial/ethnic minority groups. These proportions 
were virtually unchanged since prior to conversion (84.4 percent for students from outside 
the neighborhood and 83.0 percent for students from inside it). The share of students who 
were economically disadvantaged rose for both groups of students, which is not what was 
expected according to theory for the students from outside the neighborhood. Moreover, 
after conversion the two groups were similar (7 1 .2 percent for students from outside the 
neighborhood and 73.8 percent for students from inside; increases from 65.4 and 70.1, 
respectively).^"^ Thus, the characteristics of students from outside the neighborhood did not 
change in the direction that would be expected based on the theory of action. 


After conversion, the difference between students from outside the neighborhood and neighborhood students in 
the proportion from disadvantaged backgrounds (2.6 percentage points) was not statistically significant (p=0.072). 

The gap between students from the neighborhood and from outside the neighborhood did not change significantly 
from before to after conversion, in terms of either race/ethnicity (a 1.4 percentage point difference before conversion 
to a 0.5 percentage point difference after conversion) or economic disadvantage (a -4.7 percentage point difference 
to a -2.6 percentage point difference). That is, the characteristics of the students from outside the neighborhood were 
not “more differenf ’ from the neighborhood students after conversion than they were beforehand. 
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Exhibit 5.2. 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of students from outside the neighborhood who were from 
minority raclal/ethnic groups In the magnet schools (84.4 percent) was 1.4 percentage points higher than the average proportion of 
neighborhood students from minority racial/ethnic groups (83.0 percent), which was not a statistically significant difference. From 
pre- to post-conversion, there was no statistically significant change in the proportion of students from outside the neighborhood 
who were from minority groups (0.0 percentage points) or the proportion of neighborhood students who were from minority groups 
(1 .0 percentage points). As a result, after (post-) conversion, the average proportion of students from outside the neighborhood 
who were from minority racial/ethnic groups in the magnet schools (84.4 percent) was 0.5 percentage points higher than the 
average proportion of neighborhood students who were from minority racial/ethnic groups (84.0 percent), which was still not a 
statistically significant difference. 

NOTE: # = Rounds to zero. See Appendix F for methods used for calculations and significance tests. For the proportion of 
students from minority racial/ethnic backgrounds, N = 17 schools in 10 districts; for the proportion of students from economically 
disadvantaged backgrounds, N = 11 schools in seven districts. Results include neighborhood students and students from outside 
the neighborhood only; students missing neighborhood status were not included in this exhibit. Differences were calculated using 
unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix Exhibit 1.2 for 
additional detail. 

SOURCE: District administrative data. 
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■ After conversion, the academic achievement of students from outside the 

neighborhood in traditional magnet schools was not higher than the achievement of 
neighhorhood students (Exhibit 5,3), In ELA, both neighborhood students and students 
from outside the neighborhood scored at about the 44th percentile after conversion, with 
the test scores of both groups higher by a similar amount compared to before the 
conversion. In mathematics, both groups of students scored higher after conversion, at 
about the 48th percentile. The increase in test scores after conversion for students from 
outside the neighborhood is consistent with what theory predicts for traditional magnet 
schools. However, it is not possible with the available data and study design to determine 
whether the higher achievement for non-neighborhood students was caused by the 
recruitment of students who came with higher test scores or caused by higher levels of 
learning while in the magnet schools. In any case, the students from the neighborhood 
were not exposed to higher achieving peers from outside the neighborhood after 
conversion. 


The gaps in achievement between students from the neighborhood and from outside the neighborhood did not 
change significantly from before to after conversion, in terms of either ELA (a -1.5 percentage point difference 
before conversion to a -0.4 percentage point difference after conversion) or mathematics (a -4.1 percentage point 
difference to a 0.0 percentage point difference, p=0.091). That is, the achievement of the students from outside the 
neighborhood was not “more differenf ’ from the neighborhood students’ achievement after conversion than it was 
beforehand. It is also important to note that the test scores of students from outside the neighborhood after 
conversion could reflect differences in the academic performance of students recruited to the school before versus 
after conversion. Or the scores could reflect increases or decreases in learning among those students from outside 
the neighborhood who were attending the traditional magnets both before and after the conversion. 
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Exhibit 5.3. Achievement of Neighborhood Students and Students From Outside the 
Neighborhood in Traditional Magnet Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average ELA achievement of students from outside the neighborhood 
(35.6 percentile) was 1.5 percentage points lower than the average ELA achievement of neighborhood students (37.1 percentile), 
which was not a statistically significant difference. From pre- to post-conversion, there was a 7.3 percentage point Increase In the 
average ELA achievement of students from outside the neighborhood and a similar Increase (8.4 percentage points) In the average 
ELA achievement of neighborhood students. As a result, after (post-) conversion, the average ELA achievement of students from 
outside the neighborhood (44.0 percentile) was 0.4 percentage points lower than the average ELA achievement of neighborhood 
students (44.4 percentile), which was still not a statistically significant difference. 

NOTE: See Appendix F for methods used for calculations and significance tests. A/ = 17 schools in 10 districts. Differences were 
calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix 
Exhibit 1.3 for additional detail. 

SOURCE: District administrative data. 
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Relative to their districts, there was a lower concentration of students from minority 
racial/ethnic backgrounds in the traditional magnet schools after conversion, and the 
conversion may have played a role. 


Although the traditional conversion magnet schools did not attract new students who were 
significantly different from those attending from the neighborhood, the schools’ efforts to 
increase diversity must be viewed relative to changes in their districts’ demographics. In general, 
the study magnet schools must draw from the pool of students in their districts; changes in the 
pool can affect both the opportunities for the conversion schools to alter the composition of their 
student bodies and the concentration of different types of students in those schools when 
compared with their districts overall. There is evidence that the traditional magnet schools 
experienced a reduction in the concentration of students from minority racial/ethnic groups 
relative to their districts, which is what would be expected based on the theory of action. 

Describing changes in magnet schools and their districts. 

■ The proportion of minority students in the converting magnet schools did not 
change, while there was an increase in their districts, thereby reducing the 
concentration of these students in the traditional magnet schools relative to their 
districts (Exhibit 5.4), On average, neighborhood schools that converted to traditional 
magnet schools initially served a higher proportion of minority students (84.5 percent) 
relative to their districts (64.1 percent — a 20.4 percentage point difference). Although the 
districts experienced a 2.3 percentage point increase in the share of their students who 
were from racial/ethnic minority groups (from 64.1 percent to 66.3 percent^^), the 
traditional magnet schools remained virtually unchanged (84.5 percent to 84.9 percent; a 
0.4 percentage point increase). Thus, the proportion of minority students in the magnet 
schools became more like their districts after conversion (a difference of 18.5 percent) 
compared with before (20.4 percent), as predicted for this type of conversion approach. 


The value of 2.3 differs from 66.3 - 64.1 because of rounding. 
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Exhibit 5.4. Concentration of Racial/Ethnic Minority Students in Traditional Magnet 
Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of minority students In the magnet schools (84.5 percent) was 
20.4 percentage points higher than the proportion In their districts (64.1 percent). From pre- to post-conversion, there was no 
statistically significant change In the magnet schools’ proportion (0.4 percentage points) while the proportion In their districts 
Increased by 2.3 percentage points. As a result, after (post-) conversion, the magnet schools’ average proportion (84.9 percent) 
was 18.5 percentage points higher than the proportion in their districts (66.3 percent) — a significant change in the difference 
between the magnet schools and their districts of -1 .9 percentage points. This change represents a reduction in the concentration 
of minority students in the magnet schools relative to their districts. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = 17 schools in 10 districts. Results include all 
students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit 1.4 for additional detail. 

SOURCE: District administrative data. 


Investigating the role of magnet conversion. 

■ There is suggestive evidence that magnet conversion played a role in bringing the 
proportion of racial/ethnic minority students in the traditional magnet schools closer to 
their districts. The reduced concentration of minority group students in traditional 
magnet schools relative to their districts identified in Exhibit 5.4 could be caused by 
factors other than magnet conversion. Although it is not possible to rule in or out these 
other factors with certainty, given the study design, this hypothesis can be examined to 
some extent by using neighborhood schools that did not convert as a kind of proxy for 
what would be expected to happen to the study schools if they had not converted. The 
statistical approach, a comparative interrupted time series analysis, examined whether 
the schools’ student composition became more like their districts after conversion than 
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would be predicted based on changes in neighborhood schools that did not convert 
(see Appendix F).^^ 

■ Without conversion, the study magnet schools would have been predicted to differ from 
their districts by 21.7 percentage points (no change from before conversion), based on the 
changes in neighborhood schools that did not convert (Exhibit 5.5). The converting 
magnet schools’ actual difference from their districts in the proportion of minority 
students served was 19.8 percentage points, or 1.9 percentage points less than would be 
predicted, which provides support for the hypotheses that conversion, rather than other 
factors, helped bring the magnet school composition closer to their districts. 


The analysis examined whether the sehools’ student eomposition beeame more like their distriets, regardless of 
whether the magnets and other neighborhood publie sehools served a higher or lower proportion of students from 
minority raeial/ethnie groups than their distriet prior to eonversion. To do this, the differenee was measured using 
the absolute value so that a smaller gap between a sehool and its distriet meant a smaller value, regardless of where 
they started. Although, on average, the sehools that beeame traditional magnets started off serving a higher 
proportion of minority students than their distriets, two sehools started off with a lower proportion. 

The analysis shown in Exhibit 5.5 was based on the average eoneentration of minority students in magnet sehools 
and neighborhood publie sehools relative to their distriets for the years before eonversion and the average for the 
years after. An additional analysis was eondueted to examine the trend in the eoneentration of minority students 
from the first to the fourth year after eonversion, in ease ehanges took a few years to observe. Those results indieate 
that magnet sehools grew eloser to their distriets eaeh year after eonversion, relative to neighborhood publie 
sehools — eonsistent with what might be expeeted if implementation took time. See Exhibit 1.19 in Appendix I for 
results. 

Attraeting students from nonminority raeial/ethnie baekgrounds might have been ehallenging for sehools loeated 
in distriets with few nonminority students. The study eheeked the sensitivity of the results to this potential diffieulty 
by eliminating from the analysis the three traditional sehools (and eorresponding neighborhood publie sehools) that 
were in distriets that had limited raeial/ethnie diversity (greater than 80 pereent of the students in the distriet were 
from minority raeiakethnie baekgrounds). The results were similar to those with the full set of sehools. See 
Appendix H.l for more detail. 
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Exhibit 5.5. The Role of Magnet Conversion in the Concentration of Racial/Ethnic 

Minority Students in Traditional Magnet Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average size (absolute value) of the difference between the proportion of minority 
students served by the magnet schools and their districts was 21 .7 percentage points. Based on neighborhood schools that did 
not convert, If the magnet schools had not converted, the size of the predicted difference after (post-) conversion would also be 

21 .7 percentage points. Postconversion, the actual size of the difference between the magnet schools and their districts was 

1 9.8 percentage points, or 1 .9 percentage points less than predicted, a statistically significant difference associated with 
conversion. This indicates that the magnet conversion could be a factor in bringing the proportion of minority students in magnet 
schools 1 .9 percentage points closer to their districts. Statistical testing was not conducted on the preconversion-to-postconversion 
changes in this figure; only the primary outcome, the difference between the predicted and actual magnet outcome, was tested. 

NOTE: See appendix Exhibits 1.5, 1.6, and 1.7 for additional detail. 

SOURCE: District administrative data. 


Relative to their districts, the concentration of economically disadvantaged students in 
traditional magnet schools was unchanged after conversion. 


Because creating more diversity in economic background is also an aim, the analysis examined 
whether the proportion of low-income students in the study’s traditional magnet schools decreased 
relative to the district average after conversion. The data indicate that the shifts in the proportion of 
economically disadvantaged students served by the magnet schools were similar to those 
experienced by both the district overall and neighborhood schools that did not convert. Thus, the 
magnet schools did not become more like their districts. 

Describing changes in magnet schools and their districts. 

■ The share of students who were from low-income backgrounds in the converting 
schools increased rather than decreased, with the change reflecting a districtwide 
change (Exhibit 5,6), Neighborhood schools that converted to traditional magnet schools 
initially served a higher proportion of disadvantaged students (71 .4 percent) relative to their 
districts (46.1 percent; a 25.3 percentage point difference). These schools experienced an 
increase in the share of their students who were disadvanted (7 1 .4 percent to 74.2 percent, a 
2.8 percentage point change) during the conversion period, which is similar to that of their 
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districts (46.1 percent to 50.1 pereent; a 3.9 pereentage point ehange^'). Beeause the 
proportion of students who were economieally disadvantaged rose by comparable amounts 
in both the traditional magnet schools and their districts, the concentration of disadvantaged 
students in the magnet sehools did not signifieantly deeline relative to their districts as 
would be expected based on the theory of aetion for this eonversion approach. 


Exhibit 5.6. 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of economically disadvantaged students In the magnet schools 
(71 .4 percent) was 25.3 percentage points higher than the proportion In their districts (46.1 percent). From pre- to post-conversion, 
the average proportion of disadvantaged students In the magnet schools and their districts Increased by similar amounts (2.8 and 
3.9 percentage points respectively). As a result, after (post-) conversion, the magnet schools’ average proportion (74.2 percent) was 
24.1 percentage points higher than the proportion In their districts (50.1 percent) — a non-sIgnIfIcant change In the difference 
between the magnet schools and their districts of -1 .2 percentage points. While the magnet schools were 1 .2 percentage points 
closer to their districts after conversion, this change is not statistically different from zero and, thus, does not represent a reduction in 
the concentration of disadvantaged students in the magnet schools. 

NOTE: See Appendix F for methods used for calculations and significance tests. A/ = 1 1 schools in seven districts. Results include 
all students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit 1.8 for additional detail. 

SOURCE: District administrative data. 


Investigating the role of magnet conversion. 

■ There is not evidence that the magnet conversion influenced the concentration of 
economically disadvantaged students in traditional magnet schools. To assess 
whether the changes identified in Exhibit 5.6 might have been caused by magnet 
conversion, the changes were compared with the changes in neighborhood schools that 
did not convert. Without conversion (based on the changes in neighborhood schools that 


The value of 3.9 differs from 50.1 - 46.1 beeause of rounding. 

The proportion of eeonomieally disadvantaged students in traditional magnet sehools was 1.2 pereentage points 
eloser to the proportion in their distriets after eonversion, but this ehange was not statistieally signifieant. 
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did not convert), the study magnet sehools would be predieted to differ from their 
districts by 26.7 percentage points in the proportion of low-income students served 
(Exhibit 5.7). Although they actually experienced a postconversion difference from their 
distriets of 24.1 percentage points, or 2.5 pereentage points less than would be predicted, 
this ehange assoeiated with the conversion is not statistieally different from zero.^^ 


Exhibit 5.7. The Role of Magnet Conversion in the Concentration of Economically 

Disadvantaged Students in Traditional Magnet Schools (Average Across 
Schools) 
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EXHIBIT READS: Before (pre-conversion), the average size (absolute value) of the difference between the proportion of 
disadvantaged students served by the magnet schools and their districts was 25.3 percentage points. Based on neighborhood 
schools that did not convert, if the magnet schools had not converted, the size of the predicted difference after (post-) 
conversion would be 26.7 percentage points. Instead of increasing, the average size of the difference decreased after conversion 
(1.2 percentage points), bringing them 2.5 percentage points closer to their districts than would be predicted (an actual average 
difference of 24.1 percentage points between the magnet schools instead of the 26.7 percentage points predicted). While the 
magnet schools were 2.5 percentage points closer to their districts postconversion than predicted, this difference is not statistically 
different from zero and, thus, the conversion is not likely a factor in bringing the proportion of disadvantaged students in magnet 
schools closer to their districts. Statistical testing was not conducted on the preconversion to postconversion changes in this figure; 
only the primary outcome, the difference between the predicted and actual magnet outcome, was tested. 

NOTE: See appendix Exhibits 1.9, 1.10, and 1.1 1 for additional detail. Differences were calculated using unrounded numbers and 
may not equal differences calculated from numbers shown on the figure. 

SOURCE: District administrative data. 


The analysis shown in Exhibit 5.7 was based on the average concentration of disadvantaged students in magnet 
schools and neighborhood public schools relative to their districts for the years before conversion and the average 
for the years after. An additional analysis was conducted to examine the trend in the concentration of disadvantaged 
students from the first to the fourth year after conversion, in case changes took a few years to observe. However, 
those results indicate that magnet schools did not grow closer to their districts each year after conversion, relative to 
neighborhood public schools. See Exhibit 1.19 in Appendix I for results. 
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Average achievement in the traditionai magnet schoois was higher after conversion, 
outpacing district improvement in ELA but not in mathematics; however, there is not 
evidence that the conversion piayed a roie in improving the achievement gains of 
neighborhood students, the popuiation of poiicy interest. 


The theory of action for traditional magnet schools suggests multiple ways that achievement 
might be improved. Attracting higher achieving students from outside the neighborhood after 
conversion could raise average test scores in the schools simply because these students’ higher 
achievement pushes up the average, or because of spillover effects on the lower achieving 
neighborhood students (e.g., setting higher standards for behavior or academic motivation, 
encouraging higher expectations among teachers). If the adoption of a specialized curriculum or 
instructional methods leads to a strengthened academic program, it also could improve 
achievement for all students in the schools — beyond any improvement that their districts 
experienced overall. 

Though achievement in the traditional magnet schools improved relative to their districts, there is 
not evidence that the conversion was a factor. 

Describing changes in magnet schools and their districts. 

■ ELA achievement improved in both traditional magnet schools and their districts, 
improving more in magnet schools than in their districts (Exhibit 5.8). The average 
ELA achievement in traditional magnet schools was higher after conversion than 
before by 8.1 percentile points (average achievement increased from the 35.5th to the 
43.6th percentile). The average ELA achievement in the districts also was higher 
after conversion by 5.6 percentile points. Thus, the increase in the ELA achievement 
in the magnet schools outpaced the increase in ELA achievement in their districts by 

2.5 percentile points (8.1 percentile point increase in the magnet schools versus a 

5.6 percentile point increase in their overall districts). 

■ Mathematics achievement improved in both traditional magnet schools and their 
districts, by about the same amount (Exhibit 5.8). The average mathematics 
achievement in traditional magnet schools was higher after conversion than before by 
10 percentile points (average achievement increased from the 37.6th percentile to the 
47.6th percentile). The average mathematics achievement in the districts also was higher 
after conversion by 8 percentile points, but it was about 2 percentile points lower than the 
improvements among the traditional magnet schools. Although this 2 percentile point 
difference is in the predicted direction, it is not statistically significant. Therefore, it 
cannot be concluded with a sufficient degree of certainty that the difference is real. 


Student achievement percentiles were computed based on the first year of data, which was prior to conversion, 
separately within each district. This made it possible to assess whether scores increased or decreased across time 
relative to the base year. For example, if the median score in a district’s first year of data was 300, then a score of 
300 was standardized as the 50th percentile. If achievement generally increased above 300, then the districtwide 
median percentile would rise above 50; if achievement fell below 300, the districtwide median percentile would fall 
below 50. See Appendix F.1.1 for details on the standardization. 

The value of 5.6 differs from 56.8 - 51.1 because of rounding. 
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Exhibit 5.8. Achievement in Traditional Magnet Schools and Their Districts (Average 
Across Schools) 
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EXHIBIT READS: The average ELA achievement In the magnet schools Increased by 8.1 percentile points from before (pre-) 
conversion (35.5 percentile) to after (post-) conversion (43.6 percentile). The average ELA achievement In their districts Increased 
by 5.6 percentile points from pre-conversion (51.1 percentile) to post-conversion (56.8 percentile). Therefore, the magnet schools 
increased 2.5 percentile points more than their districts (an 8.1 percentile point increase in the magnet schools compared to a 
5.7 percentile point increase in their districts) — a statistically significant change. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = 17 schools in 10 districts. Differences were 
calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix 
Exhibit 1.12 for additional detail. 

SOURCE: District administrative data. 
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Investigating the role of magnet conversion. 

■ There is not evidence that the conversion played a role in improving the 

achievement of neighborhood students attending the traditional magnet schools. As 

was the case in assessing changes in diversity as the schools converted to become magnet 
schools, it is possible that the changes in achievement in these schools were caused by 
factors in the districts other than the conversion itself, including districtwide 
improvement efforts. In addition, it is possible that changes in the mix of students 
attending the magnet schools (i.e., different students attending the magnet schools after 
conversion) could have resulted in changes in school achievement. To help isolate the 
possible influence of conversion and rule out both district improvement efforts and 
student compositional factors, year-to-year achievement gains for students in magnet 
schools were compared to the gains that would be predicted had their school not 
converted. Predictions were based on the gains of similar students in neighborhood 
schools that did not convert. Because the policy interest is in neighborhood students, 
who tend to be lower achieving and also the largest group of students served by 
traditional magnet schools, this analysis focused solely on neighborhood student 
achievement. 

In neither ELA nor mathematics was the change in average annual achievement gains for 
neighborhood students in traditional magnet schools significantly larger or smaller than 
what would be predicted based on changes for neighborhood students in neighborhood 
schools that did not convert. If the schools they attended had not converted, the 
neighborhood students in traditional magnet schools would be predicted to have an 
annual achievement gain of 0.85 percentile points, based on the annual achievement gain 
of neighborhood students in neighborhood schools that did not convert. Although their 
actual ELA annual gain was -0.45 percentile points, 1.30 percentile points lower than 
predicted, this difference associated with conversion was not statistically different from 
zero (Exhibit 5.9). In mathematics, the neighborhood students in study schools would be 
predicted to have an annual achievement gain of 0.57 percentile points if their schools did 
not convert. Their actual annual gain was 0.65 percentile point, 0.08 percentile points 
higher than predicted, but also not statistically different from zero (Exhibit 5.10).^^ The 
similarity of observed and predicted achievement gains for neighborhood students in the 
study schools makes it unlikely that the conversion was a factor. This finding may seem 
inconsistent with the results shown in Exhibit 5.8, which indicate that the average 
achievement in magnet schools increased significantly relative to their districts. One 
hypothesis that might explain this is that the magnet schools might have attracted 
academically stronger neighborhood students after conversion, who otherwise would 


For details about the approach, a comparative interrupted time series analysis, see Appendix F.4.2.2. 

The analysis shown in Exhibits 5.9 and 5.10 were based on the average ELA and mathematics achievement gains 
of students in magnet schools and neighborhood public schools for the years before conversion and the average for 
the years after. An additional analysis was conducted to examine the trend in achievement from the first to the fourth 
year after conversion in case improvements or declines took a few years to see. Flowever, those results indicate that 
the neighborhood students’ annual ELA and mathematics achievement gains in magnet schools did not change over 
the four years post-conversion, relative to neighborhood students in neighborhood public schools. See Exhibit 1.19 
in Appendix I for results. 
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have attended other school choice options. This could have raised the average 
achievement level of neighborhood students but not their average annual gains if they 
were already high achieving when they entered magnet schools. 


Exhibit 5.9. The Role of Magnet Conversion in ELA Achievement for Neighborhood 
Students in Traditional Magnet Schools (Average Across Schools) 
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EXHIBIT READS: The average neighborhood student annual percentile point gain In ELA achievement In magnet schools before 
(pre-) conversion was 0.97 percentile points. Based on similar students In neighborhood schools that did not convert, the average 
annual percentile point gain for neighborhood students In the magnet schools would be predicted to be 0.85 percentile points after 
(post-) conversion if the magnet schools had not converted (a 0.12 percentile point decrease from before conversion). The average 
neighborhood student annual percentile point gain in ELA achievement in the magnet schools after (post-) conversion was -0.45 
percentile points (a 1.42 percentile points decrease from before conversion). Thus, the annual percentile point gain for 
neighborhood students in magnet schools was 1.30 percentile points less than would be predicted had the schools not converted. 
The change associated with conversion (-1 .30) is not statistically different from zero and, thus, there is insufficient evidence to 
conclude that conversion is a factor in changes to the achievement gains of neighborhood students. Statistical testing was not 
conducted on the preconversion to postconversion changes in this figure; only the primary outcome, the difference between the 
predicted and actual magnet outcome, was tested. This Exhibit Reads statement is only intended to walk the reader through the 
exhibit. 

NOTE: See appendix Exhibits 1.13, 1.14, and 1.15 for additional detail. 

SOURCE: District administrative data. 
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Exhibit 5.10. The Role of Magnet Conversion in Mathematics Achievement for 

Neighborhood Students in Traditional Magnet Schools (Average Across 
Schools) 
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EXHIBIT READS: The average neighborhood student annual percentile point gain in mathematics achievement in magnet schools 
before (pre-) conversion was -1 .46 percentile points. Based on similar students in other neighborhood schools, the average annual 
percentile point gain for neighborhood students in the magnet schools would be predicted to be 0.57 percentile points after (post-) 
conversion (a 2.03 percentile point increase from before conversion). The average neighborhood student annual percentile point 
gain in mathematics achievement in the magnet schools after (post-) conversion was 0.65 percentile points (a 2.08 percentile 
point increase from before conversion). Thus, the annual percentile point gain for neighborhood students in magnet schools was 
0.08 percentile points more than would be predicted had the schools not converted. The change associated with conversion (0.08) 
is not statistically different from zero and, thus, there is insufficient evidence to conclude that conversion is a factor in changes to the 
achievement gains of neighborhood students. Statistical testing was not conducted on the preconversion to postconversion changes 
in this figure; only the primary outcome, the difference between the predicted and actual magnet outcome, was tested. This Exhibit 
Reads statement is only intended to walk the reader through the exhibit. 

NOTE: See appendix Exhibits 1.16, 1. 17, and 1.18 for additional detail. 

SOURCE: District administrative data. 
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Chapter 6: Outcomes for Destination Conversion 
Magnet Schoois 

Although there were few destination magnet sehools in the study, understanding whether they 
experienced the changes in composition and achievement that the theory of action predicts is as 
important as it is for schools that became traditional magnet schools. In contrast to the 
traditional magnet schools, destination magnet schools typically begin as high-performing 
schools serving higher proportions of economically advantaged or nonminority students than 
other schools in their districts. A destination school’s new curricular theme is expected to ahract 
minority, economically disadvantaged, or low-achieving students who might then benefit from 
higher achieving peers. It is also expected to lead to improved learning for students ahending 
from both inside and outside the neighborhood. 

In this study, the schools designated as destination magnet schools were higher performing and 
more economically advantaged than their districts, but they served similar proportions of 
students from minority racial/ethnic groups. Thus, their ability to become more racially or 
ethnically diverse — that is, to become more like their district — is limited. This chapter examines 
the outcomes for the four study schools that followed this approach to conversion. 


The destination magnet schoois served a somewhat iarger share of students from 
outside their neighborhood after conversion, and these students were different from the 
neighborhood students in ways consistent with theory. 


The composition of the schools that converted to become destination magnet schools changed in 
ways that are consistent with the theory of action. After conversion, the share of students from 
outside the neighborhood increased and students from outside the neighborhood were more 
likely to be from minority racial/ethnic groups, from low-income families, and academically 
lower performing than students attending as their neighborhood schools. 

■ Even before the schools converted, about one-third (34,2 percent) of their 

enrollment came from students who lived outside the neighborhood (Exhibit 6,1), It 

may be that, as hypothesized in Chapter 5, the amount of outside enrollment reflects the 
role of choice in the districts in which the destination magnet schools are located. This 
could include the “transfer” requirements under the Elementary and Secondary Education 
Act, whereby students from consistently low-performing schools are allowed to transfer 
to higher performing ones. 


Because the number of destination magnet schools in the study sample (four) is smaller than the number of 
traditional magnets (17), it is harder to draw reliable conclusions about whether conversion was associated with 
outcomes for destination schools. See Appendix sections F.2.2.3.1, F.3.2.1, and F. 4. 2. 1.1 for additional information 
on the smallest effects the study can detect for traditional and destination magnet schools. 

In other neighborhood schools that did not convert to become magnets, 27.0 percent of their students were from 
outside the neighborhood. 
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■ For the converting schools, the share of students who were from outside the 

neighborhood rose hy 7,3 percentage points (Exhibit 6,1), Across the study schools, 
the proportion of students from outside the neighborhood inereased from 34.2 percent 
before conversion to 41.4 percent afterward. This equates to an additional 36 students 
from outside the neighborhood per converting magnet (i.e., approximately 1 additional 
student on average in eaeh class). 


Exhibit 6.1. Share of Students From Outside the Neighborhood in Destination Magnet 
Schools (Average Across Schools) 



Preconversion 


Post conversion 


EXHIBIT READS: Before (pre-) conversion, the average proportion of students from outside the neighborhood In the magnet 
schools was 34.2 percent. This proportion Increased by 7.3 percentage points to 41 .4 percent after (post-) conversion. 

NOTE: See Appendix F for methods used for calculations and significance testing. N = four schools In three districts. See appendix 
Exhibit J.1 for additional detail. 

SOURCE: District administrative data. 

■ After conversion, larger shares of students from outside the neighborhood than 
from the neighborhood were members of minority racial/ethnic groups and were 
economically disadvantaged (Exhibit 6,2), After conversion, 77.1 percent of outside 
students were from minority groups, compared to 74.6 percent of neighborhood students 
and, 35.6 percent of outside students were disadvantaged compared to 29.7 of 
neighborhood students. Therefore, compared to neighborhood students, students from 


The estimate of 1 outside student per class was derived using methods similar to the methods used for the 
analysis of traditional magnets (described in footnote 27 on page 29). The average attendance per grade in 
destination schools in our analysis (83.0 students per grade) was multiplied by 6 to estimate the average attendance 
in grades K-5 (500 students per school). The percentages in Exhibit 6.1 were then used to estimate the number of 
additional students from outside the neighborhood to be 36 (171 students before conversion and 207 after). 

Because information on the number of classes in each grade and school was not available, it was assumed that the 
average class size was 20 students (Digest of Education Statistics 2013). Based on this assumption, the increase in 
students from outside the neighborhood per class was estimated to be approximately 1 student per class (36 extra 
students over approximately 25 classes). 
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outside the destination magnet sehools’ neighborhood were 2.5 pereentage points more 
likely to be from minority raeial/ethnie groups after eonversion and 5.9 pereentage points 
more likely to be eeonomieally disadvantaged. In eontrast, before eonversion, there was 
no signifieant differenee between the neighborhood students and those from outside the 
neighborhood in terms or raee/ethnicity or eeonomie disadvantage. The differenees in the 
eharaeteristies of the two groups of students after eonversion, eombined with the higher 
proportion of outside students attending, suggest that the destination magnet sehools 
beeame more diverse, as expected based on the theory of action. 


The gap between students from the neighborhood and from outside the neighborhood did not change significantly 
from before to after conversion, in terms of either race/ethnicity (a 2.7 percentage point difference before conversion 
to a 2.5 percentage point difference after conversion) or economic disadvantage (a -1.0 percentage point difference 
to a 5.9 percentage point difference). That is, the characteristics of the students from outside the neighborhood were 
not “more differenf ’ than the neighborhood students after conversion than they were beforehand. 
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Exhibit 6.2. Characteristics of Neighborhood Students and Students From Outside the 
Neighborhood in the Destination Magnet Schools (Average Across 
Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of students from outside the neighborhood who were from 
minority raclal/ethnic groups In the magnet schools (73.6 percent) was 2.7 percentage points higher than the average proportion of 
neighborhood students from minority racial/ethnic groups (70.9 percent), which was not a statistically significant difference. From 
pre- to post-conversion, the change in the proportion of students from outside the neighborhood who were from minority groups 
(3.5 percentage points) was not statistically significant but the change in the proportion of neighborhood students who were from 
minority groups (3.7 percentage points) was. After (post-) conversion, the average proportion of students from outside the 
neighborhood who were from minority racial/ethnic groups in the magnet schools (77.1 percent) was 2.5 percentage points higher 
than the average proportion of neighborhood students who were from minority racial/ethnic groups (74.6 percent), which was a 
statistically significant difference. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Results include 
neighborhood students and students from outside the neighborhood only; students missing neighborhood status were not included 
in this exhibit. Differences were calculated using unrounded numbers and may not equal differences calculated from numbers 
shown on the figure. See appendix Exhibit J.2 for additional detail. 

SOURCE: District administrative data. 
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■ After conversion, the academic achievement of students from outside the 

neighborhood in destination magnet schools was lower than the achievement of 
neighhorhood students in mathematics hut similar in ELA (Exhibit 6,3), Destination 
schools are expected to reach out to students from lower performing schools or who are 
lower performing academically themselves. This part of the theory of action was evident 
among the study schools specifically for mathematics: after conversion, students from 
outside the destination magnet schools’ neighborhood had mathematics test scores that 
were 5.0 percentile points lower than those of the neighborhood students. But in ELA, 
neighborhood students and students from outside the neighborhood did not differ 
significantly in average scores after conversion. Before conversion, students from outside 
the neighborhood also scored lower than neighborhood students in mathematics but not in 
ELA. Because the magnet schools served a higher proportion of students from outside the 
neighborhood after conversion, and in mathematics these students were lower achieving 
than neighborhood students, the destination magnet schools gave more students from 
outside the neighborhood exposure to higher achieving peers. 


The gaps in achievement between students from the neighborhood and from outside the neighborhood did not 
change significantly from before to after conversion, in terms of either ELA (a -3.3 percentage point difference 
before conversion to a -2.0 percentage point difference after conversion) or math (a -8.8 percentage point difference 
to a -5.0 percentage point difference). That is, the achievement of the students from outside the neighborhood were 
not “more differenf ’ from the neighborhood students after conversion than they were beforehand. It is also important 
to note that the test scores of students from outside the neighborhood after conversion could reflect differences in the 
academic performance of students recruited to the school before versus after conversion. Or they could reflect 
increases or decreases in learning among those students from outside the neighborhood who were attending the 
destination magnets both before and after the conversion. 
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Exhibit 6.3. Achievement of Neighborhood Students and Students From Outside the 
Neighborhood in the Destination Magnet Schools (Average Across 
Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average ELA achievement of students from outside the neighborhood 
(55.7 percentile) was 3.3 percentage points lower than the average ELA achievement of neighborhood students (59.1 percentile), 
which was not a statistically significant difference. From pre- to post-conversion, there was a 2.5 percentile point increase in the 
average ELA achievement of students from outside the neighborhood and a similar increase (1.2 percentile points) in the 
average ELA achievement of neighborhood students. As a result, after (post-) conversion, the average ELA achievement of 
students from outside the neighborhood (58.2 percentile) was 2.0 percentage points lower than the average ELA achievement 
of neighborhood students (60.2 percentile), which was still not a statistically significant difference. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Differences 
were calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See 
appendix Exhibit J.3 for additional detail. 

SOURCE: District administrative data. 
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Relative to their districts, there was no change in the concentration of students from 
minority racial/ethnic backgrounds in the destination magnet schools, and there is not 
evidence that the conversion was a factor. 


The destination magnet schools attracted students from outside the neighborhood with 
characteristics consistent with the theory of action. This section turns to a comparison of 
destination magnet schools with their districts, focusing on the racial/ethnic diversity of the 
entire school (neighborhood and outside students). The results indicate that the magnet schools 
experienced changes that were similar to those in the district, as well as to the neighborhood 
schools in the districts that did not convert. This suggests that the conversion process probably 
had no special influence on the racial/ethnic diversity in the magnet schools. 

Describing changes in magnet schools and their districts. 

■ The proportion of minority students increased in both destination schools and their 
districts after conversion, leaving the concentration of minority students in the 
magnet schools unchanged relative to their districts (Exhibit 6.4), On average, 
neighborhood schools that converted to destination magnet schools initially served a 
proportion of minority students (71.6 percent) that was similar to the proportion in their 
districts overall (72.0 percent). Both the destination schools and their districts 
experienced an increase in the share of students from minority racial/ethnic groups 
(an increase of 3.3 percentage points for destination schools and 1.8 for their districts). 
Contrary to what would be expected, magnet schools did not become more like their 
districts after conversion. The gap between the magnet schools and their districts before 
(0.4 percentage point) versus after (1.1 percentage points) the conversion was virtually 
unchanged. 
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Exhibit 6.4. Concentration of Racial/Ethnic Minority Students in the Destination Magnet 
Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of minority students In the magnet schools (71.6 percent) 
was 0.4 percentage points lower than the proportion In their districts (72.0 percent). From pre- to post-conversion, there was a 
statistically significant Increase In the magnet schools' proportion (3.3 percentage points) and In the proportion in their districts 
(1 .8 percentage points). As a result, after (post-) conversion, the magnet schools’ average proportion (74.9 percent) was significantly 
higher (1.1 percentage points) than the proportion in their districts (73.8 percent), but the change in the difference between the magnet 
schools and their districts (0.7 percentage points) was not significant. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Results include 
all students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit J.4 for additional detail. 

SOURCE: District administrative data. 


Investigating the role of magnet conversion. 

■ There is not evidence indicating that conversion to a magnet played a role in the 
concentration of racial/ethnic minority students in destination magnet schools/^ 

The results in Exhibit 6.4 indicate that destination magnet schools maintained their 
concentration of racial/ethnic minority students relative to their districts. While this 
suggests that conversion did not play a role, it is possible that the concentration of 
minority students in neighborhood schools that did not convert did change. If the change 
in destination magnet schools differed from what would be predicted if the schools had 
not converted, this might suggest a role for the conversion in maintaining the 
concentration of minority students in the magnet schools relative to their districts, though 
other factors could also be influences. Based on the changes in neighborhood schools that 


The analysis examined whether the schools’ student composition became more like their districts, regardless of 
whether the magnets and other neighborhood schools served a higher or lower proportion of students from minority 
racial/ethnic groups than their district prior to conversion. To do this, the difference was measured using the absolute 
value so that a smaller gap between a school and its district meant a smaller value, regardless of where the school 
started. 
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did not convert, the magnet sehools would be predicted to differ from their distriets by 
4.4 percentage points if they had not converted (Exhibit 6.5). Although the magnet 
schools differed from their districts by 3.3 percentage points after conversion and, henee, 
were 1 . 1 pereentage points eloser than predicted to their districts in the proportions of 
minority group students served, that 1 . 1 differenee is not statistieally different from 


Exhibit 6.5. The Role of Magnet Conversion in the Concentration of Students From 
Minority Racial/Ethnic Backgrounds in Destination Magnet Schools 
(Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average size (absolute value) of the difference between the proportion of 
minority students served by the magnet schools and their districts was 4.0 percentage points. Based on neighborhood schools 
that did not convert, the average size of the difference between magnet schools and their districts would be predicted to Increase 
to 4.4 percentage points after (post-) conversion had the magnet schools not converted. Instead of an Increase, the magnet schools 
experienced a decrease after conversion (0.7 percentage points), bringing them 1.1 percentage points closer to their districts than 
would be predicted (a 3.3 percentage point average difference between the magnet schools and their districts instead of 4.4). 

While the magnet schools were 1.1 percentage points closer to their districts post-conversion than predicted, this difference is not 
statistically different from zero and, thus, the conversion is not likely a factor in bringing the proportion of minority students in magnet 
schools closer to their districts. Statistical testing was not conducted on the preconversion to postconversion changes in this figure; 
only the primary outcome, the difference between the predicted and actual magnet outcome, was tested. 

NOTE: See appendix Exhibits J.5, J.6, and J.7 for additional detail. 

SOURCE: District administrative data. 


The analysis shown in Exhibit 6.5 was based on the average eoneentration of minority students in magnet sehools 
and neighborhood publie sehools relative to their distriets for the years before eonversion and the average for the 
years after. An additional analysis was eondueted to examine the trend in the eoneentration of minority students 
from the first to the fourth year after eonversion, in ease ehanges took a few years to observe. However, those results 
indieate that magnet sehools did not grow eloser to their distriets eaeh year after eonversion, relative to 
neighborhood publie sehools. See Exhibit J.13 in Appendix J for results. 

The study eheeked the sensitivity of the results to the potential diffieulty of attraeting more students from minority 
raeial/ethnie baekgrounds by eliminating from the analysis the one destination sehool (and eorresponding 
neighborhood publie sehools) that was in a distriet that had few minority students (less than 20 pereent of the 
students in the distriet). However, the results were similar to those with the full set of sehools. See Appendix H.l for 
more detail. 
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Relative to their districts, the concentration of economically disadvantaged students in 
the destination magnet schools was higher after conversion, but there is not evidence 
that conversion was a factor. 


Consistent with the theory of action for destination magnet schools, the growth in the proportion 
of students from economically disadvantaged backgrounds in the converting magnet schools 
outpaced their districts. Neighborhood schools that did not convert experienced changes that 
differed from the magnet schools, but the difference did not meet conventional statistical 
benchmarks. 

Describing changes in magnet schools and their districts. 

■ The proportion of economically disadvantaged students in the converting 

destination schools increased more than in their districts (Exhibit 6,6), Neighborhood 
schools that converted to destination magnet schools initially served a smaller proportion 
of disadvantaged students than their districts (25.0 percent compared with 38.1 percent, a 
13.1 percentage point difference). The destination magnet schools experienced an 
increase in the share of their students who were disadvantaged (25.0 percent to 
31.8 percent, a 6.8 percentage point increase), compared with a smaller increase in their 
districts (38.1 percent to 41.1 percent, a 3.0 percentage point increase). Because the 
proportion rose by a greater amount for the destination magnet schools than their 
districts, the concentration of disadvantaged students relative to their districts increased in 
the destination magnet schools as would be expected based on the theory of action. 
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Exhibit 6.6. Concentration of Students From Economically Disadvantaged 

Backgrounds in the Destination Magnet Schools (Average Across Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average proportion of economically disadvantaged students In the magnet schools 
(25.0 percent) was 13.1 percentage points lower than the proportion in their districts (38.1 percent). From pre- to post-conversion, 
the average proportion of disadvantaged students increased significantly in the magnet schools (6.8 percentage points) and their 
districts (3.0 percentage points). As a result, after (post-) conversion, the magnet schools’ average proportion (31 .8 percent) was 
9.3 percentage points lower than the proportion in their districts (41.1 percent), which was a significant change in the difference 
between the magnet schools and their districts of -3.8 percentage points. This change represents a significant increase in the 
concentration of disadvantaged students in the magnet schools relative to their districts. 

Note: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Results include all 
students: neighborhood students, students from outside the neighborhood, and students missing neighborhood status. District 
proportion is based on students in all schools in the district. Differences were calculated using unrounded numbers and may not 
equal differences calculated from numbers shown on the figure. See appendix Exhibit J.8 for additional detail. 

SOURCE: District administrative data. 


Investigating the role of magnet conversion. 

■ There is not evidence that conversion influenced the concentration of economically 
disadvantaged students in destination magnet schools. To assess whether the changes 
identified in Exhibit 6.6 might have been caused by magnet conversion, the changes were 
compared with the changes in neighborhood schools that did not convert. After 
conversion, the magnet schools moved in a different direction than would be predicted 
(Exhibit 6.7). Without conversion, the study magnet schools would be predicted to differ 
from their districts by 15.6 percentage points, based on the changes in neighborhood 
schools that did not convert. The converting magnet schools’ actual difference from their 
districts in the proportion of disadvanted students served was 9.9 percentage points, or 
5.7 percentage points less than would be predicted. This difference is just short of meeting 
the benchmark most studies use to determine that a finding is reliable and not due to chance 
(i.e., the difference has a />-value of 0.066 rather than a value less than 0.05).^^ It is also the 


The study checked the sensitivity of the results to the potential difficulty of attracting more economically 
disadvantaged students by eliminating from the analysis the one destination school (and corresponding 
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case that the influenee of eonversion on the eoneentration of disadvantaged students in 
destination magnet sehools was greater in the later years after conversion than the earlier 
years, suggesting some promise about these magnet school outcomes."^’ 

Exhibit 6.7. The Role of Magnet Conversion in the Concentration of Economically 

Disadvantaged Students in Destination Magnet Schools (Average Across 
Schools) 
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EXHIBIT READS: Before (pre-) conversion, the average size (absolute value) of the difference between the proportion of 
disadvantaged students served by magnet schools and their districts was 13.6 percent. Based on neighborhood schools that did 
not convert, the average size of the difference after (post-) conversion would be predicted to Increase to 15.6 percentage points 
had the magnet schools not converted. Instead of an Increase, the magnet schools experienced a decrease after conversion 
(3.7 percentage points), bringing them 5.7 percentage points closer to their districts than would be predicted (a 9.9 percentage point 
average difference between the magnet schools and their districts Instead of 15.6). While the magnet schools were 5.7 percentage 
points closer to their districts post-conversion than predicted, this difference Is not statistically different from zero and, thus, the 
conversion is not likely a factor in bringing the proportion of disadvantaged students in magnet schools closer to their districts. 
Statistical testing was not conducted on the preconversion to postconversion changes in this figure; only the primary outcome, the 
difference between the predicted and actual magnet outcome, was tested. 

Note: See appendix Exhibits J.9, J.10, and J.11 for additional detail. 

SOURCE: District administrative data. 


neighborhood public schools) that was in a district that had few disadvantaged students (less than 20 percent of the 
students in the district). With this destination magnet excluded, there was a signifrcant relationship between the 
change in the concentration of disadvantaged students in the destination magnets relative to their districts and 
magnet conversion. The estimated change associated with conversion was -7.82 percentage points (p = 0.007). See 
Appendix H.l. 

The analysis shown in Exhibit 6.7 was based on the average concentration of disadvantaged students in magnet 
schools and neighborhood public schools relative to their districts for the years before conversion and the average 
for the years after. An additional analysis was conducted to examine the trend in the concentration of disadvantaged 
students from the first to the fourth year after conversion, in case changes took a few years to observe. The results 
indicate that magnet schools grew closer to their districts each year after conversion, relative to neighborhood public 
schools, consistent with what might be expected if implementation took time. See Exhibit J. 13 in Appendix J for 
results. 
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Achievement in destination magnet schoois did not change after conversion, whereas 
achievement in their districts improved. 


The theory of action predicts that destination magnet schools might improve achievement 
through either the specialized curriculum, which could benefit all students, or the opportunity for 
the outside students to have higher performing or more advantaged peers. However, if the 
magnet schools bring in more lower-performing students, the average achievement of the schools 
could decline. 

Describing changes in magnet schools and their districts. 

■ The destination magnet schools lost ground to their districts in ELA achievement. 

Before conversion, average ELA achievement in destination magnet schools 
(58.8th percentile) was higher than the average in their districts (51.6th percentile) — a 
difference of 7.2 percentile points (Exhibit 6.8). Average ELA achievement in the 
districts increased after conversion by 6.9 percentile points (to the 58.5th percentile), 
whereas achievement in the magnet schools did not change significantly (1.4 percentile 
point increase to the 60.2th percentile). 

■ The changes in mathematics achievement in the magnet schools and their districts 
were similar."^^ Before conversion, the average mathematics achievement in the 
destination magnet schools (58.4th percentile) was higher than the average in their 
districts (51.6th percentile) — a difference of 6.8 percentile points (Exhibit 6.8). Average 
mathematics achievement in the districts increased after conversion by 8.9 percentile 
points (to the 60.5th percentile), whereas the achievement in the magnet schools did not 
change (the 3.9 percentile point increase to the 62.2th percentile was not statistically 
different from zero)."^^ The 1.7 percentile point mathematics achievement gap between 
the magnets and their districts after conversion is just short of being statistically different 
from the 6.8 percentile gap before conversion (/7=0.051 rather than /»=0.050). 

Investigating the role of magnet conversion. 

Examining the role of conversion by comparing the changes in achievement gains for the 
destination magnet schools to those of neighborhood schools that did not convert is not 
appropriate for students from outside the neighborhood, as discussed in Chapter 2. There are 
better approaches, including the use of lotteries, to examine the effects of attending a converted 
magnet for these students. However, these approaches were not feasible in this study. 


The estimated ehange in the differenee (-5.1 pereentile points) was just short of statistieally signifieant (p=0.051 
rather than /?=0.050). 

The value of 3.9 differs from 62.2 - 58.4 beeause of rounding. 
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Exhibit 6.8. Achievement in Destination Magnet Schools and Their Districts (Average 
Across Schools) 
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EXHIBIT READS: The average ELA achievement In the magnet schools did not significantly change (Increase of 1 .4 percentile 
points) from before (pre-) conversion (58.8 percentile) to after (post-) conversion (60.2 percentile). The average ELA achievement in 
their districts increased significantly by 6.9 percentile points from preconversion (51 .6 percentile) to postconversion (58.5 percentile). 
Therefore, the districts increased 5.5 percentile points more than the magnet schools (a 1.4 percentile point increase in the magnet 
schools compared to a 6.9 percentile point increase in their districts) — a statistically significant change. 

NOTE: See Appendix F for methods used for calculations and significance tests. N = four schools in three districts. Differences were 
calculated using unrounded numbers and may not equal differences calculated from numbers shown on the figure. See appendix 
Exhibit J.12 for additional detail. 

SOURCE: District administrative data. 
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Chapter 7: Variation in Resuits and Next Steps in 
Understanding Magnet Schooi Outcomes 

This study tracked the student body composition and test scores of a set of sehools over 7 years, 
examining how they changed from before to after the schools converted to beeome magnet 
schools. Because student body composition and performance in the magnet sehools’ districts 
(and shifts in these factors) could affect the converting magnet sehools’ ability to aehieve 
improvements in diversity and achievement, ehanges in the converting magnet sehools were 
eompared to those in their distriets overall. In addition, to examine whether what oeeurred in the 
magnet schools differed from the general patterns experieneed by other distriet schools, changes 
in the magnet sehools were compared to what would be predicted if they had not eonverted 
(based on neighborhood sehools that did not convert). Two different approaches to magnet 
conversion (traditional and destination) were examined, separately, beeause the types of students 
they sought to attract and influence were different. 

The results of these analyses are mixed. Both the traditional and destination magnet schools 
experienced some ehanges in diversity in the direetion expeeted from the theory of how each 
type of conversion would play out. While the achievement in traditional magnet sehools was 
higher after eonversion, beyond their districts’ improvement in aeademic performanee, the 
destination magnet schools lost ground to their districts. However, for only one of these 
outcomes — the reduced concentration of minority students in traditional magnet schools relative 
to their districts — ^was there evidence that the conversion could have played a role, as opposed to 
other changes happening in sehools distrietwide. 

Sometimes examining the outcomes for a set of sehools on average, as was the case in the 
previous ehapters, can mask important differences among them. Not only were there two 
categories of magnet schools — traditional and destination — but they were implemented in 
different ways and in different eontexts. Looking at the individual results for each converting 
magnet eompared to what would be predieted had it not eonverted ean help determine whether a 
small subset of the sehools have a big influence on the results or whether the results are 
eonsistent across schools in the study. A lot of variation, in particular, may suggest that 
eonverting magnet schools or their eontexts may be so different that an average look aeross them 
(even separately by traditional versus destination) may be less meaningful, or at least that it 
would be important to study a larger number of these schools in the future. Consisteney in the 
outeome comparisons across individual schools would suggest that the average results from this 
study may be a good marker of what other converting magnet sehools might experience. 

Conducting this type of analysis reveals significant variation in how individual magnet schools 
changed in comparison to what would be predieted based on neighborhood sehools that did not 
convert — i.e., whether eonversion may have a role in the changes seen in diversity or 
aehievement. For example, in 9 of the 17 traditional magnet sehools, eonversion may have been 
a factor in an increase in racial/ethnie diversity. (See Exhibit 7.1, which shows that nine bars 
were signifieantly below the horizontal line, indieating that the differenee between those schools 
and their distriets was reduced.) In another three schools the eonversion may be related to a 
decrease in raeial/ethnic diversity, and in five sehools there is not evidenee to suggest any role at 
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all (the changes in outcomes for the magnet schools were similar to the changes that would be 
predicted). Among the 1 1 traditional magnet schools with data on student disadvantage, 
conversion may have been a factor in an increase in economic diversity in 6 traditional magnet 
schools and a decrease in economic diversity in 2 schools. In the remaining three schools, there 
is not evidence of conversion playing a role. 

In terms of achievement, neighborhood students in some traditional magnet schools (seven in 
ELA and six in mathematics) fared better than would be predicted based on similar students in 
neighborhood schools in their districts that did not convert, while some other traditional magnet 
schools (five in ELA and nine in mathematics) fared worse (Exhibit 7.2). In the four destination 
schools, conversion may have been a factor in increased economic diversity in three of these 
schools. In terms of racial/ethnic diversity, outcomes were split (Exhibit 7.3).^" 

Ideally, this variation would provide an opportunity to assess whether specific aspects of the 
magnet schools’ implementation or context are related to better or worse outcomes. 
Unfortunately, there were too few converting schools in the study sample to draw even tentative 
hypotheses about what factors might contribute to that variation. Eor example, there were only 
4 schools that used a STEM focus for their magnet schools — not enough to determine whether a 
STEM focus was related to better or worse outcomes than other conversion themes. Euture 
research on magnet conversion should make such an assessment a priority. 


Tests for significant variation across schools in the association between conversion and outcomes were conducted 
and results indicated significant variation in each instance tested (see Appendices I and J, Exhibits 1.6, 1. 10, 1.14, 
1.17, J.6, and J.IO). See Appendix F.2.2.3 for more details. 
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Exhibit 7.1. The Role of Magnet Conversion in the Concentration of Racial/Ethnic 

Minority and Economically Disadvantaged Students in Traditional Magnet 
Schools 



-15 


■ Minority ■ Disadvantaged 


EXHIBIT READS: Each pair of bars represents results for one traditional magnet school. From before (pre-) conversion to after 
(post-) conversion, the far left traditional magnet school experienced a 2.5 percentage point decrease in the difference between the 
proportion of racial/ethnic minority students served by this magnet school and its district beyond what would be predicted based on 
the changes in neighborhood schools in this district that did not convert (indicated by the light grey bar). The same magnet school 
experienced a 2.4 percentage point decrease in the difference between the proportion of disadvantaged students served by this 
magnet school and its district beyond what would be predicted based on the changes in neighborhood schools in this district that did 
not convert (indicated by the dark grey bar). 

NOTE: N = M study magnet schools in 10 districts for the percentage minority; A/ = 1 1 study magnet schools in 7 districts for the 
percentage disadvantaged. (3 districts G, J, and K did not have data on whether students were disadvantaged.) A statistically 
significant estimate is denoted by the following symbol: * = statistically significant with p < .05, two-tailed test. 

SOURCE: District administrative data. 


65 


What Happens When Schools Become Magnet Schools? 


Exhibit 7.2. The Role of Magnet Conversion in Achievement for Neighborhood Students 
in Traditional Magnet Schools 

30 
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EXHIBIT READS: Each pair of bars represents results for one traditional magnet school. From before (pre-) conversion to after 
(post-) conversion, the average neighborhood student annual percentile point gain in ELA achievement in the far left traditional 
magnet increased by 5.3 percentile points more than would be predicted based on the average neighborhood student annual 
percentile point gain in ELA achievement in neighborhood schools that did not convert (indicated by the light grey bar). From before 
(pre-) conversion to after (post-) conversion, the average neighborhood student annual percentile point gain in mathematics 
achievement in the far left traditional magnet decreased by 1 .9 percentile points more than would be predicted based on the 
average neighborhood student annual percentile point gain in mathematics achievement in neighborhood schools that did not 
convert (indicated by the dark grey bar). 

NOTE: N = 17 turnaround schools in 10 districts. A statistically significant estimate is denoted by the following symbol: 

* = statistically significant with p < .05, two-tailed test. 

SOURCE: District administrative data. 
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Exhibit 7.3. The Role of Magnet Conversion in the Concentration of Racial/Ethnic 

Minority and Economically Disadvantaged Students in Destination Magnet 
Schools 
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EXHIBIT READS: Each pair of bars represents results for one destination magnet school. Over the conversion period, the far left 
destination magnet school experienced a 0.9 percentage point decrease In the difference between the proportion of raclal/ethnic 
minority students served by this magnet school and Its district beyond what would be predicted based on the changes In 
neighborhood schools in this district that did not convert, which was not a statistically significant difference (indicated by the light 
grey bar). The same magnet school experienced a 0.1 percentage point increase in the difference between the proportion of 
disadvantaged students served by this magnet school and its district beyond what would be predicted based on the changes in 
neighborhood schools in this district that did not convert, which was also not a statistically significant difference (indicated by the 
dark grey bar). 

NOTE: N = four study magnet schools in three districts. A statistically significant estimate is denoted by the following symbol: 

* = statistically significant with p < .05, two-tailed test. 

SOURCE: District administrative data. 
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Appendix A: Districts and Schoois in the Study 

A.1. Sample Selection 

The magnet eonversion sample for this study eonsists of 21 schools in 1 1 distinct districts 
selected from the 2004 and 2007 MSAP grant cohorts, the two most recent cohorts at the time 
the study was initiated. (Three schools in two districts^' were selected from the 2004 cohort, and 
18 schools in 10 districts, one of which was also included in the first cohort, were selected from 
the 2007 cohort.) Sample selection began with a full list of 385 grantee magnet schools (in 
81 districts) from the 2004 and 2007 funding cohorts. Data provided by the MSAP program 
office were used to narrow the potential sample to 1 19 schools (in 41 districts) in which existing 
elementary schools were to be converted into schoolwide magnet schools. Because the main 
analyses to be conducted for the study involved comparing achievement before and after magnet 
school conversion, schools in states where the state student assessments were not consistent for 
two years prior to and two years after the magnet conversion were eliminated. Finally, the study 
collected additional information by mail and phone from district MSAP directors for the 
remaining 46 candidate schools in 24 districts to apply two final selection criteria. The following 
additional information was sought: 

■ Magnet schools with stable attendance boundaries during the study period^^ 

■ Districts that were able to provide the data necessary to conduct the analyses presented 
here, including information to distinguish between neighborhood students and students 
from outside the neighborhood in each school in the district^"^’^^ 


One district contained study magnet schools in both the 2004 and 2007 cohorts. 

Since the initiation of this study, MSAP has funded new cohorts of districts in 2010 and 2013. 

Stable attendance zones were needed to allow for identification and comparison of resident and nonresident 
students in a manner that did not change across time. Unstable zones would cause changes in the resident and 
possibly nonresident populations, making it difficult to infer patterns related to the magnet school conversion itself 

The original study design proposed to use a sample of matched comparison schools. Thus, one additional criterion 
used to select the magnet study sample was the availability of one or more matching comparison schools in the 
magnet school’s district. In the final analysis, however, all noncharter and nonmagnet schools in the district were 
used as comparisons. Restricting the sample to MSAP districts with at least one matched comparison school for each 
selected magnet school resulted in the omission of one district with one magnet conversion school that might have 
been kept in the study. The district was eliminated because the closest potential matching school had test scores and 
demographics very different from the magnet conversion school’s demographics (greater than a 0.5 standard 
deviation difference in test scores; greater than a 20 percentage point difference in percentage minority and 
percentage free or reduced-price lunch eligible). For details on the decision to use all noncharter and nonmagnet 
schools in the district as comparisons rather than creating a matched comparison sample, see Betts et al. (2009), 
which describes a simulation study conducted as part of the design phase of this research project. This paper is 
available on request from the authors. 

It is possible that districts that were able to provide the data and thus were included in the study were different 
from those not included in ways that also affected the magnet school conversion process. For example, the districts 
that were included may have had more sophisticated data systems and thus perhaps may also have had more 
experience adopting new practices than districts not included. 
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The 21 magnet sehools in 1 1 distriets from the 2004 and 2007 eohorts that met these eriteria 
eonstituted the study analysis sample. Given the restrietions applied in the seleetion proeess, 
this sample of sehools and distriets was purposive; it was not a representative or random sample 
of the population of MSAP-funded elementary magnet sehools, funded distriets, or sehools 
within funded distriets. 

A.2. Information on State Assessments 

One eriterion for distriet seleetion listed previously was that the district’s state student 
assessments were consistent for at least two years prior to and two years after the magnet 
conversion. As part of site selection for the study conducted in early 2007, information was 
collected on changes in state assessments that had occurred through the 2006-07 school year and 
changes anticipated after the 2006-07 school year. The information was collected from technical 
documentation on the assessment websites of all states containing potential study districts. 

Changes that were considered major and to have interrupted consistency were the following: 
change from fall to spring testing, changes in content standards, and the use of new assessments. 
Changes that were considered minor and not to have interrupted consistency included the 
following: changes in cut scores for percent proficient and changes in the month of the 
assessment within the same school year (e.g., from January to April). 

Although districts were eliminated during prescreening if there were major changes in their state 
assessments, a subsequent review in 2013 of documentation on state assessment websites 
indicated that in a few districts, changes had occurred outside the years screened. Exhibits A. 1 
and A.2 indicate for each district and each analysis year whether there was no major change in 
the assessment (indicated by v^), or if there was a change (indicated by A). In District C, the 
change in ELA and mathematics assessments was a shift from fall to spring testing. The other 
changes in Districts D, E, and 1 in 2010-1 1 were either changes in content standards or the use of 
new assessments. 


Twenty-two magnet sehools were analyzed for the study, but two sehools from the 2007 eohort were loeated in 
the same building and had been one sehool prior to eonversion. These two eo-loeated sehools were treated as one 
sehool in the analysis, resulting in 21 sehools. 
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Exhibit A.1. Years of Assessment Data Used, Base Year for Standardization for Chapter 6 Analysis, 
and Changes for ELA Assessments 


Districts (Cohorts) 

Number of Study 

Schooi Year 

Magnet Schoois 
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4 








•/ 

— 

C (2007) 

1 

— 

— 





A 

•/ 


D (2007) 

3 

— 

— 

— 



y' 

•/ 

■/ 

A 

E (2007) 

1 

— 

— 

— 

— 

•/ 



•/ 


F (2007) 

3 

— 

— 

— 

— 

•/ 





G (2007) 

2 

— 

— 

— 







H (2007) 

1 

— 

— 

— 







I (2007) 

1 

— 

— 

— 


•/ 




A 

J (2007) 

2 

— 

— 

— 







K (2007) 

2 

— 

— 

— 


•/ 

•/ 





NOTE: — = Not available; data were not collected or not reported. = no change in assessment from the previous year. ^ = change in assessment from the previous year. 
SOURCE: Documentation from state assessment websites. 
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Exhibit A.2. Years of Assessment Data Used, Base Year for Standardization for Chapter 6 Analysis, 
and Changes for Mathematics Assessments 


Districts (Cohort) 

Number of Study Magnet 

Schooi Year 

Schoois 

2002-03 

2003-04 

2004-05 

2005- 

o 

O) 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

A (2004) 

1 









— 

B (2004 and 2007) 

4 









— 

C (2007) 

1 

— 

— 


•/ 



A 

■/ 


D (2007) 

3 

— 

— 

— 






A 

E (2007) 

1 

— 

— 

— 

— 




■/ 


F (2007) 

3 

— 

— 

— 

— 




■/ 


G (2007) 

2 

— 

— 

— 

•/ 




■/ 


H (2007) 

1 

— 

— 

— 

•/ 






1 (2007) 

1 

— 

— 

— 






A 

J (2007) 

2 

— 

— 

— 







K (2007) 

2 

— 

— 

— 



•/ 





NOTE: — = Not available; data were not collected or not reported, •^ = no change in assessment from the previous year, A = change in assessment from the previous year. 
SOURCE: Documentation from state assessment websites. 
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A.3. Grades and Years Analyzed 

Exhibit A.3 provides the grades and school-years analyzed for each district, as well as the 
number of preconversion and postconversion years included in the analysis, and the total number 
of student observations. 


Exhibit A.3. MSAP Grant Year, Grades, and Years Analyzed for Study Districts 


District 

MSAP 
Grant Year 

Grades 

Anaiyzed 

Years 

Anaiyzed 

Number of 
Years 

Preconversion 

Number of 
Years 

Postconversion 

Number of 
Student 
Observations 

A 

2004 

3-5 

2002-03 to 
2007-08 

2 

4* 

450,224 

B (2004) 

2004 

2-5 

2002-03 to 
2007-08 

2 

4* 

44,867 

B (2007) 

2007 

2-5 

2002-03 to 
2009-10 

5 

3 

48,053 

C 

2007 

2-4 

2004-05 to 
2010-11 

3 

4 

11,507 

D 

2007 

3-5 

2005-06 to 
2010-11 

2 

4 

186,195 

E 

2007 

3-5 

2006-07 to 
2010-11 

1 

4 

164,521 

F 

2007 

4-5 

2006-07 to 
2010-11 

1 

4 

10,857 

G 

2007 

3-5 

2005-06 to 
2010-11 

2 

4 

134,549 

H 

2007 

2-5 

2005-06 to 
2010-11 

2 

4 

51,124 

I 

2007 

3-5 

2005-06 to 
2010-11 

2 

4 

150,048 

J 

2007 

2-5 

2005-06 to 
2010-11 

2 

4 

221,027 

K 

2007 

3-5 

2005-06 to 
2010-11 

2 

4 

42,776 

TotaF 






1,470,881 


t Total row includes numbers from District B (2007) but not District B (2004); * Study schools from the 2004 MSAP grant year for 
which data were also available for two additional years: the 2008-09 and 2009-10 school years. These extra years of data were 
included in statistical models that analyzed achievement to increase precision of parameter estimates for the covariates, but 
reported results for magnet conversion pertain only to the first four postconversion years. 

SOURCE: District administrative data. 
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A.4. Characteristics of the Study Districts 


Chapter 2 reported comparisons of the districts selected for the study to all districts that received 
MSAP support and all districts in the United States. Exhibits A.4 and A. 5 report the results for 
district size and geographic location, which were discussed in Chapter 2 but not presented in 
exhibits in the chapter. 


Exhibit A.4. Distribution of Districts by Size of Study Districts, All MSAP Districts, and 
All U.S. Public School Districts 


(0 

O 

b 


o 

<1> 

O) 

(0 


0 > 

o 

<D 

Q_ 



Number of Students in the District 


■ Study Districts All MSAP Districts DAII U.S. Public School Districts 


* All study districts significantly different from all U.S. public school districts (p < .05); t All MSAP districts significantly different from 
all U.S. public school districts (p < .05). 

NOTE: A/ = 12 for sample districts with District B represented twice in the data, once for the 2004 and once for the 2007 MSAP 
funding cycles. For all MSAP districts and all districts nationally, distributions represent a weighted average of 2003-04 and 2006- 
07 distributions, where the 2003-04 numbers were weighted 2/12 for the two 2004 funding cycle study districts represented in the 
exhibit, and the 2006-07 numbers were weighted 10/12 for the ten 2007 funding cycle districts represented in the exhibit. 

SOURCE: CCD 2003-04 and 2006-07. 


One study district containing magnet schools from both funding cycles is represented twice in background 
exhibits to display the differing context for the 2004 and 2007 funding cycle magnet schools in this district. Hence, 
the count of districts here, and in other exhibits when noted, is 12 rather than 1 1 . 
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Exhibit A.5. Region of Study Districts, All MSAP Districts, and All U.S. Public School 
Districts 



Region 


■ Study Districts 

■ All MSAP Districts 

□ All U.S. Public School Districts 


t All MSAP districts significantly different from all U.S. public school districts (p < .05). 

NOTE: N= 12 for sample districts with District B represented twice In the data, once for each of the 2004 and 2007 MSAP funding 
cycles. For all MSAP districts and all districts nationally, the percentages In the exhibit are a weighted average of the 2003-04 
and 2006-07 percentages, where the 2003-04 percentages were weighted 2/12 for the two 2004 funding cycle study districts 
represented in the exhibit, and the 2006-07 percentages were weighted 10/12 for the ten 2007 funding cycle districts represented 
in the exhibit. 

SOURCE: Common Core of Data 2003-04 and 2006-07. 


A.5. Classification of Study Schools as Traditional or Destination 
Magnet Schools 

As described in Chapter 2, study magnet schools were classified as traditional or destination 
based on the following criteria. 

■ A school was classified as a traditional magnet school if, prior to conversion, its average 
achievement in ELA and in mathematics were both below the district average, and the 
proportion of students in the school from minority backgrounds and the proportion from 
disadvantaged backgrounds were both above the proportion for the district as a whole. 
Fourteen schools met all four of these criteria. Three additional schools that met three of 
the four criteria were also classified as traditional.^^ 

■ A school was classified as a traditional magnet school if, prior to conversion, its average 
achievement in ELA and in mathematics were both below the district average, and the 
proportion of students in the school from minority backgrounds and the proportion from 
disadvantaged backgrounds were both above the proportion for the district as a whole. 


The percentage of students from economically disadvantaged backgrounds was not available for six schools. On 
the basis of their other characteristics, all six were classified as traditional magnet schools. 
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Fourteen sehools met all four of these criteria. Three additional schools that met three of 
the four criteria were also classified as traditional. 

Exhibit A.6 provides the magnitude of the difference between the magnet school and the district. 
These differences in averages were calculated across all preconversion years and used as the 
basis of the classification of schools. Exhibit A.6 also indicates whether each school was above 
the district average in ELA and mathematics achievement (“yes” if above, “no” if below); and 
whether it was below the district in the proportion of students from minority racial/ethnic and 
disadvantaged backgrounds (“yes” if below, “no” if above). 


Exhibit A.6. Ciassification of Study Schools as Traditional or Destination Magnet 
Schools 


District 

MSAP 

Grant 

Year 

Schooi 

iD 

Number 

Difference From District 
Average Achievement 
(Schooi - District) & 
Whether Above the 
District Average (Yes/No) 

Difference From District 
Proportion 
(Schooi - District) & 
Whether Beiow the 
District Proportion 
(Yes/No) 

Magnet 

Type 




ELA 

Mathematics 

Minority 

Disadvantaged 


A 

2004 

001 

-0.233 

No 

0.164 

Yes 

7.2% 

No 

18.4% 

No 

Traditional 

B 

2004 

001 

0.026 

Yes 

0.124 

Yes 

-1 .2% 
Yes 

0.5% 

No 

Destination 

B 

2004 

002 

-0.467 

No 

-0.474 

No 

1 .7% 
No 

15.3% 

No 

Traditional 

B 

2007 

003 

-0.107 

No 

-0.109 

No 

-1.1% 

Yes 

15.8% 

No 

Traditional 

B 

2007 

004 

0.455 

Yes 

0.479 

Yes 

1 .8% 
No 

-32.9% 

Yes 

Destination 

C 

2007 

001 

-0.435 

No 

-0.586 

No 

26.1% 

No 

14.8% 

No 

Traditional 

D 

2007 

001 

-0.725 

No 

-0.564 

No 

36.2% 

No 

42.3% 

No 

Traditional 

D 

2007 

002 

-0.610 

No 

-0.583 

No 

29.3% 

No 

25.1% 

No 

Traditional 

D 

2007 

003 

-0.752 

No 

-0.572 

No 

39.3% 

No 

43.9% 

No 

Traditional 

E 

2007 

001 

0.104 

Yes 

0.202 

Yes 

5.3% 

No 

-17.9% 

Yes 

Destination 

F 

2007 

001 

0.102 

Yes 

-0.113 

No 

-7.6% 

Yes 

-2.3% 

Yes 

Destination 

F 

2007 

002 

-0.551 

No 

-0.803 

No 

27.3% 

No 

31.3% 

No 

Traditional 

F 

2007 

003 

-0.226 

No 

-0.127 

No 

23.1% 

No 

11.8% 

No 

Traditional 
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District 

MSAP 

Grant 

Year 

Schooi 

iD 

Number 

Difference From District 
Average Achievement 
(Schooi - District) & 
Whether Above the 
District Average (Yes/No) 

Difference From District 
Proportion 
(Schooi - District) & 
Whether Beiow the 
District Proportion 
(Yes/No) 

Magnet 

Type 

G 

2007 

001 

-0.468 

No 

-0.288 

No 

17.8% 

No 

N/A 

Traditional 

G 

2007 

002 

-0.437 

No 

-0.238 

No 

43.1% 

No 

N/A 

Traditional 

H 

2007 

001 

-0.696 

No 

-0.562 

No 

22.0% 

No 

27.0% 

No 

Traditional 

1 

2007 

001 

-0.345 

No 

-0.725 

No 

43.7% 

No 

32.8% 

No 

Traditional 

J 

2007 

001 

-0.265 

No 

-0.261 

No 

-9.7% 

Yes 

N/A 

Traditional 

J 

2007 

002 

-0.213 

No 

-0.161 

No 

5.8% 

No 

N/A 

Traditional 

K 

2007 

001 

-0.084 

No 

-0.032 

No 

19.8% 

No 

N/A 

Traditional 

K 

2007 

002 

-0.294 

No 

-0.355 

No 

15.3% 

No 

N/A 

Traditional 


NOTE: N/A indicates that the category is not appiicabie. Three districts, Districts G, J, and K, did not have data to identify 
economicaiiy disadvantaged students. To be ciassified as a destination magnet schooi, a schooi needed at ieast one "Yes” in 
“Difference From District Average Achievement” (i.e., above the district in ELA or mathematics or both) and at ieast one yes in 
“Difference From District Proportion” (i.e., beiow the district proportion in minority or disadvantaged or both). 

SOURCE: District administrative data. 
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Appendix B: Use and Coding of Administrative Data 

The analyses reported in Chapters 5 and 6 and Exhibits 3.4 and 3.5 were based on administrative 
data collected from participating study districts. The information requested and received from 
each study district was student-level data for multiple years before and after magnet conversion 
(typically two years before and four years after conversion; see Exhibit A. 3 for details on years 
collected and analyzed) and included the following types of variables: 

■ Student Achievement, Standardized test scores in ELA and mathematics from the state’s 
annually administered standardized achievement tests 

■ Student Characteristics, Grade, age, race or ethnicity, gender, free or reduced-price 
lunch eligibility (except from Districts G, J, and K^^), special education status, and 
English language learner (ELE) status 

■ Student’s school of enrollment 

■ Student’s catchment school (i.e., assigned neighborhood school) 

The four sections in this appendix discuss the use and coding of these administrative data. Eirst, 
the structure of the data elements provided by the districts is presented. This section focuses on 
enrollment and catchment school data. Second, the definition and coding of student residency 
status are discussed. Residency status determinations were based on district-provided enrollment 
and catchment school data. Third, the definition and coding of school dosage and test-to-test 
years are presented. The final section reports sample sizes for achievement analyses. 

B.1. Structure of Administrative Data 

Data on the four types of variables listed previously were obtained at least once per year, with 
some data provided at multiple time points. 

■ Achievement, Achievement data were recorded once per year, at the time of annual 
achievement testing. 

■ Student Characteristics, Student characteristics were recorded by districts once per year. 

■ Student’s School of Enrollment, Each student’s school of enrollment (the school in 
which the student was enrolled) was recorded at least once per year. In district-provided 
data, a student’s school of enrollment was typically indicated by the school’s building ID 
code. Enrollment data files came in two basic forms: transactional and snapshot. As 
shown in the third column of Exhibit B.l, five districts provided transactional data, and 
six districts provided snapshot data. 


District J did not provide free or reduced-price lunch eligibility information for its students. Districts G and K 
provided free or reduced-price lunch eligibility information that was judged to be unreliable. Specifically, at least 
one study magnet school in these districts had a large change (greater than 30 percentage points) in free or reduced- 
price lunch eligibility across time. For these schools, the district-provided percentages for free or reduced-price 
lunch eligibility were compared with the statistics for the same school years reported in the Common Core of Data; 
these large changes were not reflected there. 
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• Transactional Data. Transactional data are structured as one observation per student, 
per year, per “event,” with starting and ending dates. An event is generally a change 
in the school of enrollment, and the period between the starting and ending dates is an 
enrollment “spell.” If a student was enrolled in only one school in a given year, he or 
she would have one observation for that year. If a student was enrolled in more than 
one school in a given year, the student would have one observation for each school in 
which he or she was enrolled during that year, and each observation would contain 
the start and end dates for the enrollment spell in that school. 

• Snapshot Data. In contrast, snapshot data are structured as one enrollment status 
record for each student in a district on a particular date. Snapshot data are an extract 
of enrollment data on all students in a district, typically taken on a few dates each 
school year. Multiple snapshots for each school year were collected from six districts 
that used this type of enrollment data: Five districts provided two snapshots per year, 
and one district provided three snapshots per year. Thus, for districts that provided two 
snapshots per year, the study typically had two observations for each student, for each 
year the student was enrolled in the district.^' Students who changed schools in the 
period between snapshots would have different schools of enrollment in their two 
snapshot records. Unlike transactional data, snapshot data do not provide the exact dates 
that a student changed schools; the analysis imputed these dates based on the timing of 
the snapshots. The method of imputation was to divide the time between snapshots in 
half and assign half of the time to each of a student’s observed enrollment spell. 

■ Student’s Catchment School. A student’s catchment school is the student’s assigned 
neighborhood public school — the school to which the student was assigned based on 
residence. In district-provided data, a student’s catchment school was typically indicated 
by the building ID code of his or her neighborhood school. The missing data rate for the 
catchment variable varied by district. Further, the catchment information did not always 
match the type (transactional versus snapshot) or frequency per year of the enrollment 
data. The difference in type and frequency occurred because catchment school 
information was typically derived from a different data source in the district data system 
than the enrollment school data. Catchment school information was always provided at 
least once a year. Exhibit B.l lists the study districts, the category of enrollment school 
data they provided, and the frequency of catchment school information relative to the 
enrollment data. 

• In 3 of the 1 1 districts, enrollment information and the catchment data were provided 
with equal frequency. In the “Frequency of Catchment School Data per Year” 
column, this is designated with “Simultaneous” — in other words, every enrollment 
entry had corresponding catchment school information. 

• In 8 of the 1 1 districts, the catchment school information was provided once per year. 


“ Some events not related to ehanging sehools — for instanee, a suspension or leaving the distriet for some period of 
time — eati be represented by additional transaetions within a sehool year. Sueh extraneous transaetions were 
removed during the ereation of the final analysis data fde. 

This statement is true provided the student did not leave or enter the distriet in between snapshots, in whieh ease 
there would be just one observation (i.e., present in only one snapshot) for that student. 
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Exhibit B.1. Enrollment Data Type and Frequency of Catchment Data, by District 


District 

Cohort 

Enrollment Data Type 

Frequency of Catchment School 
Data per Year 

A 

2004 

Snapshot 

Once 

B 

2004 and 2007 

Transactional 

Once 

C 

2007 

Transactional 

Simultaneous 

D 

2007 

Snapshot 

Simultaneous 

E 

2007 

Transactional 

Once 

F 

2007 

Transactional 

Simultaneous 

G 

2007 

Snapshot 

Once 

H 

2007 

Snapshot 

Once 

I 

2007 

Snapshot 

Once 

J 

2007 

Transactional 

Once 

K 

2007 

Snapshot 

Once 


NOTE: Data for Districts G and K came from the same administrative body even though they are separate districts. 
SOURCE: District administrative data. 


When catchment data were provided less often than enrollment data, the time of year the 
catchment school was recorded was taken into account when matching catchment data with 
enrollment data to establish student residency. For example, if the catchment information was 
recorded at the start of the school year, it would be merged to the first snapshot of the school 
year. When students changed schools within a school year, the catchment school was treated as 
missing in periods for which the enrollment and catchment data could not be matched in this 
way. Thus, if in the second snapshot the student had a new enrollment school, the catchment 
school indicator was set to missing. 

B.2. Residency Status Definition and imputation 

A student’s residency status was defined as an indicator of whether or not the student attended 
the assigned catchment school. If the student attended the assigned school, he or she was 
considered a neighborhood student. If not, the student was considered a student from outside the 
neighborhood. In analyses, a student’s residency status was classified as one of three values: 
neighborhood student, student from outside the neighborhood, or unknown. Unknown represents 
cases in which residency status could not be determined because of missing data. 

Districts did not provide residency status directly. Instead, it was derived from a student’s 
enrollment school ID and catchment school ID, which was defined previously. 

Residency status was then determined as follows: 

■ If a student’s enrollment school ID was the same as his or her catchment school ID, the 
student was classified as a neighborhood student. 

■ If a student’s enrollment school ID was not the same as his or her catchment school ID, 
the student was classified as a student from outside the neighborhood. 
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■ If a student’s catchment school ID was missing, he or she was classified as having an 
unknown residency status. 

B.2.1. Extent of Missing Catchment Schooi Data 

Exhibit B.2 reports the percentage of observations with missing catchment data by district and by 
year. Districts A, C, and D had catchment school information for all students. Districts G and K 
were each missing catchment data for all students in the 2005-06 school year. 
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Exhibit B.2. Percentage of Student Observations With Missing Catchment Data, by District and by Year 


District 

Cohort 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

A 

2004 

0 

0 

0 

0 

0 

0 

0 

0 

— 

B 

2004 and 
2007 

0.1 

0.1 

0.2 

# 

0.1 

0.0 

0.0 

# 

— 

C 

2007 

— 

— 

0 

0 

0 

0 

0 

0 

0 

D 

2007 

— 

— 

— 

0 

0 

0 

0 

0 

0 

E 

2007 

— 

— 

— 

— 

9.5 

9.5 

8.4 

7.8 

6.8 

F 

2007 

— 

— 

— 

— 

10.3 

10.2 

8.5 

7.9 

10.3 

G 

2007 

— 

— 

— 

100.0 

2.5 

2.4 

5.3 

5.8 

3.7 

H 

2007 

— 

— 

— 

0 

0 

0 

0 

0 

0 

1 

2007 

— 

— 

— 

0.3 

0.2 

0.2 

0.3 

0.0 

1.1 

J 

2007 

— 

— 

— 

10.1 

9.0 

9.0 

9.2 

7.6 

6.6 

K 

2007 

— 

— 

— 

100.0 

6.6 

6.6 

5.3 

8.3 

5.9 


NOTE: Data for Districts G and K came from the same administrative body even though they are separate districts. # = Rounds to zero. — = Not avaiiabie; data were not coiiected or 
not reported. 

SOURCE: District administrative data. 
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B.2.2. Imputation of Catchment School 

Given the rate of missing catchment data (see Exhibit B.2), the analysis imputed catchment 
school, and, hence, residency status, for some students using their own data in other years. In 
addition, to determine residency status for each time point for which school of enrollment data 
were available, the analysis imputed catchment data within school years wherever a district had 
provided enrollment data more frequently than catchment data. Because school of enrollment did 
not change for most students within the school year, only a small proportion of students required 
within-school-year imputations. The methods used for imputing catchment school across time 
periods are described in this section. The methods were used for both within- and between-year 
imputation. 

When information about a student’s catchment school was missing, available catchment school 
information for a given student was used to impute the student’s catchment school, using three 
specific conditions (scenarios) that are described in the following subsections. Information from 
one student was never used to impute the catchment school for another student. The primary 
condition for imputing a student’s catchment school in period (school year) T is that the student’s 
enrollment school must be the same in period T as in at least one of the adjacent periods T- 1 or 
r+i.“ The underlying logic is that if a family has not switched a child from one school to 
another, then it is unlikely that they moved out of the neighborhood, causing the child’s 
catchment school to change. 

Scenario 1. Imputation in Time JWith Information From Both Time T—\ and Time T+\ 

This scenario occurred when a student was continuously enrolled at the same school across three 
periods, and non-missing catchment school information was available from observations in two 
bracketing time periods — T~ 1 and T + 1 — to impute information at time T. Exhibit B.3 
provides the relevant data for two students: students 1 and 2. As shown for student 1, if the 
enrollment school at time T equaled the enrollment school at time T - \ and the enrollment 
school at time T+ \, and the catchment school at time T- \ equaled the catchment school at 
time T+ \, then the missing catchment school at time T was set equal to the catchment school of 
T - \ and T + 1. However, as shown for student 2, if the catchment school at time T- \ was not 
equal to the catchment school at time T + 1, then no imputation was made. 


® T is used generically to number a series of enrollment spells: T= \ might or might not be within the same sehool 
year as T=h. Henee, the examples here apply to imputing within a sehool year as well as aeross sehool years. 

It is possible that a student eould have ehanged resideney status without ehanging the sehool of enrollment. 
However, sueh a switeh does not refleet the typieal magnet sehool transition in whieh students join a sehool from 
outside its eatehment area. 
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Exhibit B.3. Example of Imputing at Time TWith Information From Both 
Time T - 1 and Time T + 1 



Student 1 

Student 2 

Tim6 

T- 1 

T 

T+ 1 

T- 1 

T 

T+ 1 

Enrollment school 

A 

A 

A 

A 

A 

A 

Catchment school 

X 

Missing 

X 

X 

Missing 

Y 

Catchment school with imputation 

X 

X 

X 

X 

Missing 

Y 


NOTE: A is an enrollment school ID; X and Y are catchment school IDs that may or may not be equal to A. 


Scenario 2, Imputation in Time TWith Information From Either Time T - 1 or Time T+\ 

When continuous enrollment or bracketing catchment information was unavailable, information 
from either the period before {T- 1) or the period after (T+ 1) was used to impute. Exhibit B.4 
illustrates the method, using two students. As illustrated by student 1, if a student’s enrollment 
school at time T equaled his or her enrollment school at time T- \, and that student’s catchment 
school at time T was missing, then the catchment school at time T was set to be equal to the 
student’s catchment school at time T - 1. As illustrated by student 2, if the enrollment school 
changes, no imputation was made. 


Exhibit B.4. Example of Imputing at Time TWith Information From Time T - 1 



Student 1 

Student 2 

Tims 

T- 1 

T 

T- 1 

T 

Enrollment school 

A 

A 

A 

B 

Catchment school 

X 

Missing 

X 

Missing 

Catchment school with imputation 

X 

X 

X 

Missing 


NOTE: A and B are enrollment school IDs; X Is a catchment school ID that may or may not be equal to A or B. 


The rationale was the same for imputing time T with information in time T+ 1, as illustrated in 
Exhibit B.5. 


Exhibit B.5. Example of Imputing at Time TWith Information From Time T + 1 



Student 1 

Student 2 

Tims 

T 

T+ 1 

T 

T+ 1 

Enrollment school 

A 

A 

A 

B 

Catchment school 

Missing 

X 

Missing 

X 

Catchment school with imputation 

X 

X 

Missing 

X 


NOTE: A and B are enrollment school IDs; X Is a catchment school ID that may or may not be equal to A or B. 


These situations occur when (1) the student did not have continuous enrollment at the same school for three 
periods; (2) catchment school information was missing for one of the bracketing periods, T - 1 or r + 1; or (3) the 
period to be imputed was either the first or last time the student was observed, and hence, data for the student were 
missing for period T- 1 or r+ 1, respectively. 
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Scenario 3, Imputation in Time J With Information From Either Time T-2 or Time T+2. 

Catchment school information was also imputed across more than one period when certain 
conditions were met. As in scenarios 1 and 2, this approach is predicated on a student’s having 
had continuous enrollment at the same school. This situation is illustrated in Exhibit B.6 for 
imputing a value in period T with information from time T+2; however, the rationale is the 
same for imputing with information from period T~ 2 (not illustrated). If a student’s enrollment 
school at time T equaled his or her enrollment school at time T+ \ and his or her enrollment 
school at time T+2 and the catchment school was missing in time T and time T+ \, both were 
set to equal the catchment school in time T+2. 


Exhibit B.6. Example of Imputing at Time T and at Time T + 1 With Information 
From Time T+2 



Student 1 


T 

T+ 1 

T+2 

Enrollment school 

A 

A 

A 

Catchment school 

Missing 

Missing 

X 

Catchment school with imputation 

X 

X 

X 


NOTE: A is an enrollment school ID; X is a catchment school ID that may or may not be equal to A. 


B.2.3. Summary of Extent of Imputation and Time Periods Used for Imputation 

In Districts A, C, D, and H, there was no need to impute residency status (see Exhibit B.2). Eor 
the remaining seven districts where imputation was used, Exhibits B.7 through B.I3 summarize 
the extent to which values were imputed and which time periods were used in the imputation. 
These exhibits display the missing rates before imputation, the percentage imputed under each 
scenario described previously, and the missing rate after imputation for each district included in 
the study. 
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Exhibit B.7. Missing Catchment Data and Imputation for District B 


District B 

imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

0.1 

0.1 

0.2 

# 

0.1 

0.0 

0.0 

# 

— 

Imputed (%) 

From T- 1 
and T + 1 

0.0 

0.0 

0.0 

# 

0.0 

0.0 

0.0 

0.0 

— 

From T- 1 
or 7 + 1 

# 

0.1 

0.1 

0.0 

# 

0.0 

0.0 

# 

— 

From 7-2 
or 7+ 2 

# 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

0.0 

— 

Missing after imputation (%) 

All 

# 

# 

0.1 

# 

# 

0.0 

0.0 

0.0 

— 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 


Exhibit B.8. Missing Catchment Data and Imputation for District E 


District E 

Imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

— 

9.5 

9.5 

8.4 

7.8 

6.8 

Imputed (%) 

From 7-1 
and 7 + 1 

— 

— 

— 

— 

# 

0.4 

0.1 

# 

0.0 

From 7-1 
or 7 + 1 

— 

— 

— 

— 

5.9 

5.3 

5.1 

4.7 

4.3 

From 7-2 
or 7+2 

— 

— 

— 

— 

0.5 

0.6 

0.4 

0.3 

0.2 

Missing after imputation (%) 

All 

— 

— 

— 

— 

3.1 

3.2 

2.8 

2.8 

2.3 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 
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Exhibit B.9. Missing Catchment Data and Imputation for District F 


District F 

imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

— 

10.3 

10.2 

8.5 

7.9 

10.3 

Imputed (%) 

From T - 1 
and T + 1 

— 

— 

— 

— 

0.0 

0.0 

0.0 

0.0 

0.0 

From T - 1 
or T + 1 

— 

— 

— 

— 

0.0 

0.0 

3.5 

2.5 

4.6 

From T-2 
or T+2 

— 

— 

— 

— 

0.0 

0.0 

0.0 

0.8 

0.4 

Missing after imputation (%) 

All 

— 

— 

— 

— 

10.3 

10.2 

5.0 

4.6 

5.3 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 


Exhibit B.10. Missing Catchment Data and Imputation for District G 


District G 

Imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

100.0 

2.5 

2.4 

5.3 

5.8 

3.7 

Imputed (%) 

From T - 1 
and T + 1 

— 

— 

— 

0.0 

0.0 

0.1 

# 

0.0 

0.0 

From T - 1 
or 7 + 1 

— 

— 

— 

44.6 

0.3 

1.0 

3.5 

4.6 

0.2 

From 7-2 
or 7+ 2 

— 

— 

— 

# 

# 

0.0 

0.1 

0.1 

2.5 

Missing after imputation (%) 

All 

— 

— 

— 

55.4 

2.2 

1.3 

1.7 

1.1 

1.1 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 
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Exhibit B.11. Missing Catchment Data and Imputation for District I 


District i 

imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

0.3 

0.2 

0.2 

0.3 

0.0 

1.1 

Imputed (%) 

From T- 1 
and T + 1 

— 

— 

— 

# 

0.0 

# 

# 

0.0 

0.0 

From T- 1 
or 7 + 1 

— 

— 

— 

# 

# 

# 

0.1 

0.0 

1.0 

From 7-2 
or 7+2 

— 

— 

— 

0.0 

# 

# 

# 

0.0 

0.0 

Missing after imputation (%) 

All 

— 

— 

— 

0.2 

0.2 

0.1 

0.1 

0.0 

0.2 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 


Exhibit B.12. Missing Catchment Data and Imputation for District J 


District J 

Imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

10.1 

9.0 

9.0 

9.2 

7.6 

6.6 

Imputed (%) 

From 7-1 
and 7 + 1 

— 

— 

— 

# 

0.1 

0.1 

0.1 

0.1 

0.0 

From 7-1 
or 7 + 1 

— 

— 

— 

4.8 

4.3 

4.4 

4.4 

3.4 

1.3 

From 7-2 
or 7+2 

— 

— 

— 

# 

# 

# 

# 

# 

# 

Missing after imputation (%) 

All 

— 

— 

— 

5.2 

4.6 

4.6 

4.6 

4.1 

5.2 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 
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Exhibit B.13. Missing Catchment Data and Imputation for District K 


District K 

imputation 
Type Used 

2002-03 

2003-04 

2004-05 

2005-06 

2006-07 

2007-08 

2008-09 

2009-10 

2010-11 

Missing prior to imputation (%) 

None 

— 

— 

— 

100.0 

6.6 

6.6 

5.3 

8.3 

5.9 

Imputed (%) 

From T - 1 
and T + 1 

— 

— 

— 

0.0 

0.0 

0.1 

0.1 

0.0 

0.0 

From T - 1 
or T + 1 

— 

— 

— 

42.0 

0.5 

3.1 

3.6 

6.8 

0.2 

From T-2 
or T+2 

— 

— 

— 

0.0 

0.0 

0.0 

0.1 

0.1 

4.0 

Missing after imputation (%) 

All 

— 

— 

— 

58.0 

6.0 

3.4 

1.6 

1.4 

1.6 


NOTE: # = Rounds to zero. — = Not available; data were not collected or not reported. 
SOURCE: District administrative data. 
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B.3. School Enrollment Dosages and Test-to-Test Years Used in 
Achievement Anaiysis That Predicted Achievement for Students in 
Magnet Conversion Schoois Based on Neighborhood Schoois in Their 
Districts That Did Not Convert 

The achievement analysis that predicted what the student achievement would have been in 
magnet conversion schools had those schools not been converted used student enrollment 
information provided at multiple time points each year when it was available. As noted in 
Chapter 2, each student’s dosage (time spent) at each school was calculated in “test-to-test 
years” — the time between administrations of the state tests. The concepts of enrollment dosage 
and test-to-test years are discussed further in this section. 

B.3.1. Enrollment Dosages 

An enrollment dosage is the proportion of a school year that a student was enrolled at a given 
school. The length of time a student was enrolled in a given school was expressed as a proportion 
of the school year — a continuous measure between 0 and 1 . 

B.3.2. Test-to-Test Years 

For the achievement analysis, years were measured as the time between each annual test 
administration date and the next (i.e., between the time points when achievement outcomes were 
measured) rather than based on the start and end of the school year. 

The unit of observation for the achievement analysis was a given student in a given year. Using 
dosages and test-to-test years allowed the analysis to associate student gains proportionately with 
each school a student attended in the period during which the achievement gain outcome was 
measured. If, for example, a student enrolled in school A took the state tests in April, attendance 
at school A in May and June cannot have had a causal impact on the test score gain observed in 
April. Further, if a student attended two schools during the year prior to the time of testing, it 
would not be valid to attribute the entire gain to only one of those schools. Using dosages and 
test-to-test years allowed the analysis to associate the observed test score gain proportionally 
with the schools each student attended in the period between tests. 

The construction of dosages and test-to-test years is described further in the following sections. 
Both dosages and test-to-test years required the following pieces of information: 

■ Date of the first and last day of the school year 

■ Date of transfer (if any) between schools 

■ Date of annual assessment 

B.3.3. Partitioning a School Year 

The enrollment dosage for a given student in a given school is the proportion of the school year 
that the student spent at the given school. 


B-13 



What Happens When Schools Become Magnet Schools? — Appendices 


Step 1. Partition School Year. To create dosages, the school year was first divided into parts 
defined by the following time points: the start of the school year, the date of transfer to a 
different school, the date of the annual assessment, and the end of the sehool year. At a 
minimum, a full sehool year was broken into two spells: 

■ From the start of the sehool year to the annual assessment date 

■ From the annual assessment date to the end of the school year 

If a student switched schools between the first day of school and the assessment, the school year 
was broken into three spells: 

■ From the start of the sehool year to the transfer date 

■ From the transfer date to the annual assessment 

■ From the annual assessment to the end of the sehool year 

For districts with transactional enrollment data, the start, transfer, and end dates could be 
determined preeisely. For distriets with snapshot enrollment data, the exact dates of any transfers 
between schools were unknown. Thus, the following assumptions were implemented: 

■ A student started the school year at the same sehool at which he or she was enrolled in 
the first snapshot of the sehool year. 

■ A student ended the sehool year at the same sehool at whieh he or she was enrolled in the 
last snapshot of the school year. 

■ If a student ehanged sehools between snapshots, the student transferred halfway between 
the dates at which the snapshots were taken. 

For districts with transactional enrollment data as well as districts with snapshot enrollment data, 
if the assessment date was not provided, it was assumed to be at the midpoint of the state 
assessment testing window. 

Step 2. Calculate Proportions for Each Partition. The proportion of the school year represented 
by eaeh part of the sehool year defined in Step 1 was caleulated. The denominator for the 
proportion was the number of days in the school year (i.e., between the first and last day of the 
school year); the numerator was the number of days in the part (e.g., number of days between the 
start of the school year and the first transfer to another school). 

Step 3. Associate Each Partition With a School of Enrollment and a Residency Status. Because 
transfer to a different sehool and change in residency status were used as time points to divide a 
school year into parts, eaeh student had only one sehool of enrollment and one resideney status 
(neighborhood student, student from outside the neighborhood, or unknown) for eaeh part of the 
school year as defined previously. A single sehool of enrollment and a single residency status 
was thus associated with each part of the year for each student. Eaeh student had multiple 
reeords for each school year, one for each part, and each record or part had an associated school 
of enrollment and residency status. 
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B.3.4. Shifting Time Frame From Schooi Years to Test-to-Test Years 

The analysis required one reeord per student per year. The proeedure to calculate dosages for 
each part of the year described previously resulted in multiple records per student per school 
year. These multiple observations per student per year were collapsed into one record per student 
per test-to-test year (i.e., from the previous year’s annual assessment to the current year’s 
assessment) rather than one record per student per school year (i.e., first day of school to last day 
of school). Both the collapsing and the definition of a test-to-test year are further described next. 

To understand the creation of a test-to-test year, consider as an example a student who never 
transferred schools during two school years: 2003-04 and 2004-05. Under the partitioning 
described previously, two partitions for each school year would be created, or four parts total: 

■ Part A = start of 2003-04 school year to assessment in 2003-04 school year 

■ Part B = assessment in 2003-04 school year to the end of the 2003-04 school year 

■ Part C = start of 2004-05 school year to assessment in 2004-05 school year 

■ Part D = assessment in 2004-05 school year to the end of the 2004-05 school year 

Hence, parts A and B together constituted the 2003-04 school year, and parts C and D 
constituted the 2004-05 school year. To shift to test-to-test years, the parts between assessments 
were associated with each other. Together, these parts between tests formed a test-to-test year. In 
this example, parts B and C together would have formed the 2004-05 test-to-test year. 

B.3.5. Assembly of Test-to-Test Year “Dosages” 

To obtain one record per student per year, parts that made up the test-to-test years were 
combined. The parts were collapsed by creating one variable for each school and each residency 
status. For example, if a district had 40 schools, 120 variables (40 schools times three potential 
residency statuses =120 school-residency variables) were created. Each school-residency 
variable (i.e., dosage) contained the proportion of the test-to-test year that a student spent at a 
given school with a given residency status. For a student who did not switch schools or change 
residency status during a test-to-test year, all the variables except one would be zero, and the 
value of the remaining variable would be 1 . However, if a student switched schools or residency 
status in the middle of the test-to-test year, more than one school-residency variable would be 
more than zero. 

B.4. Sample Sizes for Achievement Analysis 

As described in Appendix F.1.1, the outcomes for the descriptive analysis of change in 
achievement were EFA and mathematics z-scores standardized relative to the first year of data in 
the analysis data set, whereas the outcomes for the comparative interrupted time series (CITS) 
analysis on relative change in achievement used the annual z-score gain, based on z-scores 
standardized within each year. (See Appendix F.4.2 for further discussion of standardization of 
achievement scores for analysis.) Student records that did not have achievement outcomes were 
excluded from the analysis. Exhibit B.14 provides the overall sample size by district in the first 
column (which matches the number of student observations for the study sample reported in 
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Exhibit 3.6 in Chapter 3^^), sample sizes for the achievement z-seore analysis, and sample sizes 
for the student z-score gain analysis for each subject. Recall that for construction of the z-score 
gain, data were necessarily not included in the analysis sample for one year and one grade 
because these were the base scores for calculating gains. 


Exhibit B.14. Number of Student Observations, Observations With z-Scores, 
Observations With z-Score Gain for ELA and Mathematics 


District 

MSAP 

Funding 

Cycie 

Study Sampie 

Number of 
Student 
Observations 

Number of Observations 
With z-Score 

Number of Observations 
With z-Score Gain 



ELA 

Mathematics 

ELA 

Mathematics 

A 

2004 

450,224 

305,404 

305,553 

260,035 

260,215 

B (2004) 

2004 

44,867 

28,968 

29,030 

23,126 

23,199 

B (2007) 

2007 

48,053 

31,160 

31,235 

25,076 

25,163 

C 

2007 

11,507 

7,126 

7,151 

5,525 

5,525 

D 

2007 

186,195 

123,101 

123,261 

102,723 

102,937 

E 

2007 

164,521 

111,817 

112,222 

91,669 

92,076 

F 

2007 

10,857 

9,637 

9,637 

8,822 

8,822 

G 

2007 

134,549 

99,219 

102,100 

85,049 

89,649 

H 

2007 

51,124 

41,002 

40,998 

32,616 

32,594 

I 

2007 

150,048 

64,126 

64,113 

51,947 

51,900 

J 

2007 

221,027 

130,314 

130,164 

104,098 

103,884 

K 

2007 

42,776 

28,701 

29,332 

24,289 

25,195 

Total® 


1,470,881 

951,607 

955,766 

791,849 

797,960 


® Total row includes numbers from B (2007) but not B (2004). 
SOURCE: District administrative data. 


For example, in District A, the lowest grade for which achievement data were available was 
grade 2. Thus, the lowest grade for which a gain in achievement could be calculated was grade 3 
because computing the gain involved subtracting grade 2 achievement from grade 3 
achievement. The first year of achievement data available for District A was the 2001-02 school 
year. Thus, the first year for which a gain in achievement could be calculated for District A was 
the 2002-03 school year because computing the gain required subtracting grade 2 achievement 
in 2001-02 from grade 3 achievement in 2002-03. 


In this table as in Exhibit 3.6, a given student may have been eounted multiple times, onee for eaeh year he or she 
appeared in our analysis data. 
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Appendix C: Director interviews 

To obtain the data on district improvement policies and other characteristics of study magnet 
schools reported in Chapters 4 and 5, semistructured telephone interviews were conducted with 
officials in 10 of the 1 1 MSAP grant districts between July 2011 and December 2011.^^ An 
official in the 11th district returned written responses to the interview questions in February 
2012. At the time of the interviews, the MSAP grants awarded in 2007 had recently ended, and 
the districts were writing their final reports. Participants were provided with the interview 
questions several days in advance of the conversation. The interviews took one to two hours to 
complete. 

The retrospective, self-reported data requested during the interviews were intended to serve 
several purposes: 

■ To ensure that the classification of schools included in the study’s achievement analysis 
as conversion magnet schools and other neighborhood public schools during the entire 
grant period was accurate 

■ To determine the extent to which the study schools had implemented magnet programs 
resembling those described in their grant applications 

■ To provide information about the district policy context in which the programs had been 
implemented — in particular, the existence of other school reform initiatives that might 
have enhanced or reduced the contrast between the study schools and nonmagnet schools 

■ To provide information about the choice context in which the programs had been 
implemented — in particular, district officials’ perception of the extent to which the study 
schools’ ability to attract and retain students had been influenced by competition from 
other local schools with special offerings or stronger records of student achievement 

When possible, the interview was conducted with the director of the local MSAP project. When 
the individual who had directed the project during most of the grant period was no longer 
available, other district staff members who were knowledgeable about the MSAP project, the 
district school choice program, or district school improvement initiatives were consulted. 

Exhibit C.l summarizes the interview participants. 


One district received MSAP grants in both 2004 and 2007. Both grants were managed by the same project 
director; the interview for this district encompassed the study schools from both grant cohorts and local conditions 
during both grant periods. 

The two grants awarded in 2004 had been finished for four years in 201 1, but the project directors in these two 
districts were still available for interviews. Many of the topics addressed in the 2011 interview also had been 
discussed as part of the 2007 screening interviews, as described in Chapter 2 for sample selection, in which these 
same individuals had participated. The 2007 screening interviews coincided with the end of the 2004 grants. 
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Exhibit C.1. Individuals Participating in the Project Director Interviews 


District Staff Responding to the Project Director interview 

Number of Districts 

MSAP project director 

6 

School choice program director or manager 

2 

Other® 

3 

Total 

11 


® other staff included a magnet program facilitator recently promoted to the district magnet program directorship, a student 
placement director, and staff responsible for special projects and secondary school reform. 
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Appendix D: Principal Survey 

To obtain the data on study magnet school implementation reported in Chapter 5, a survey was 
administered to the principals of the conversion magnet schools in the study at (or after) the end 
of their MSAP grants. Principals were given the choice of responding to paper or electronic 
versions of the survey. To ensure confidentiality, principals were asked to return their responses 
directly to AIR using either FedEx or e-mail. Respondents were offered $25 gift certificates to 
acknowledge the time and effort required to complete the survey. 

The purpose of the survey was to obtain background information about the following: 

■ The principals and the resources, staffing, and implementation history of the magnet 
school programs during the MSAP grant period 

■ The district policy context in which the programs were implemented during that period 

Surveys were sent to the principals of three schools that were funded through 2004 MSAP grants 
during summer and fall 2007, shortly after their grants ended. Two principals completed the 
survey. The third principal had left the district at the time the survey was administered, and 
(according to the district official who collaborated with us on administering the survey) no one 
else at the school had the knowledge necessary to complete it. 

Before the survey was administered to the principals of schools funded through the 2007 MSAP 
cohort, a few questions were added to collect information about issues that had become more 
salient during the years following the first survey administration. Consequently, information on 
some topics is available only for the 19 principals whose schools were in the 2007 MSAP cohort. 

The survey of 19 principals whose schools were funded through 2007 MSAP grants began in 
fall 2010, at the beginning of the fourth year of the 2007 MSAP grant period. Nonrespondents 
were contacted by e-mail and telephone until all the principals had returned completed surveys. 
(The last surveys were returned in June 2012.)^^ 

Exhibit D.l summarizes the number of surveys administered and returned by the 22 conversion 
magnet school principals. As noted previously, two of the study schools that had been co-located 
were treated as one school for the analyses presented in this report. Flowever, for reporting 
responses from principal surveys, they are treated separately as the two principals responded 
separately. 


Exhibit D.1. Principal Surveys Distributed and Completed 


MSAP Grant Year 

Principals Surveyed 
(N) 

Responses (N) 

Response Rate 

2004 

3 

2 

67% 

2007 

19 

19 

100% 

Overall 

22 

21 

95% 


One district did not allow surveys to be distributed until June 2012. 
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Appendix E: Additional Data Sources 

To obtain contextual background and magnet program implementation information about study 
districts and schools (presented in Chapters 3 and 4), data on the district and study schools were 
collected from interviews with the MSAP directors (see Appendix C for details on the 
interviews), a survey of study magnet school principals (see Appendix D for details on the 
survey), and three sources of extant data: the Common Core of Data (CCD), EDFacts data, and 
MSAP applications (see Exhibit E.l). The CCD is a centralized data set maintained by the 
National Center for Education Statistics that contains institutional information (e.g., location, 
type, student body demographics) on schools and districts collected from state education 
agencies. The CCD does not, however, contain academic performance data. EDFacts is 
similarly a centralized repository of information on schools and districts from state education 
agencies. In contrast to the CCD, EDFacts collects academic performance and accountability 
data. Specifically, it contains school-level information on adequate yearly progress (AYP) and 
“in need of improvement” status. MSAP applications were reviewed to identify the type of 
specialized curricula and instructional methods implemented in study magnet schools. 


Exhibit E.1. Data Sources and Collection Details 


Source 

Purpose 

Information Collected and Timeline for Collection 

CCD 

Context and 
primary analysis 

■ Content. District- and school-level information for background 
comparisons 

■ Student demographic characteristics (race or ethnicity, free or 
reduced-price lunch eligibility) 

■ Identification of magnet and charter schools in study districts for 
primary analysis 

■ Sample. All U.S. elementary schools and districts, 2001-02 to 

2009- 10 school years for 2004 cohort schools, and 2004-05 to 

2010- 11 school years for 2007 cohort schools; universe data (all 
districts and study schools were represented in the data) 

■ Time Period of Data Collection. Obtained from the CCD website 

EDFacts data 

Context 

■ Content. School-level federal accountability information submitted 
by states to the Department of Education, including program 
improvement status and AYP status 

■ Sample. All elementary schools in study districts for 2003-04 to 
2009-10 school years; universe data (all study schools were 
represented in the data) 

■ Time Period of Data Collection. Obtained November 201 1 

MSAP 
applications 
for study 
districts 

Context and 
implementation 

■ Content. Information on school magnet program to be 
implemented with grant 

■ Sample. All study magnet districts and schools 

■ Time Period of Data Collection. Obtained from MSAP program in 
fall 2007 


See http://www2.ed.gov/about/inits/ed/edfacts/index.html for information describing EDFacts. 
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Appendix F: Details of Analysis Methods 

This appendix provides additional details on the methods used to answer the study’s researeh 
questions; 

1 . Did the composition of neighborhood students and students from outside the 
neighborhood in the magnet schools change after conversion? 

2. Did the diversity in the magnet schools increase after conversion, and is there any 
evidence that it was related to conversion? 

3. Did the achievement in the magnet schools increase after conversion, and is there any 
evidence that it was related to conversion? 

Overview of Methods for Research Question 1 

In both traditional and destination schools, the theory of action suggests that attracting students 
from outside the neighborhood is an important first step for converting magnet schools. Because 
it is possible that these schools could grow or shrink as a result of the movement of families in or 
out of the district, the analysis takes the size of schools into account by examining the number of 
students from outside the neighborhood as a share of the total enrollment. If conversion schools 
attracted students from outside the neighborhood as predicted, then this share should have been 
higher after conversion. 

The theory of action also predicts that the students from outside the neighborhood will differ from 
the students attending the school as their local neighborhood school. For example, traditional 
magnet schools are expected to attract nonminority, economically advantaged, higher achieving 
students; and destination magnet schools are expected to attract minority, economically 
disadvantaged, and lower achieving students. The analysis compared the characteristics of 
students from outside the neighborhood to students from inside the neighborhood in the years 
after conversion using the following measures; 

■ The proportion who were from minority racial/ethnic backgrounds; 

■ The proportion who were from economically disadvantaged backgrounds; 

■ The average ELA achievement; and 

■ The average mathematics achievement. 

If traditional magnet schools were successful in their recruitment, then after conversion the 
proportion of students from minority racial/ethnic backgrounds would be lower among students 
from outside the neighborhood than inside. The same would be expected when looking at the 
proportions of students who were from economically disadvantaged backgrounds. For ELA and 
mathematics achievement, the theory predicts that the average for students from outside the 
neighborhood would be higher than the average for neighborhood students. For destination 
schools, the predictions are reversed. 

This analysis focused on comparing the demographic characteristics and achievement of students 
from inside and outside the neighborhood after conversion, because theory predicts that 
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neighborhood students in traditional magnet sehools would have aeeess to peers who were 
different than themselves and benefit from a “spillover” effeet. Theory prediets that students 
from outside the neighborhood in destination sehools would also benefit from aeeess to higher 
achieving peers in the converted schools. Since there were students from outside the 
neighborhood already in the magnet schools prior to conversion, an additional step examined 
whether the differences between the students from outside the neighborhood and the 
neighborhood students were greater after conversion. 

Additional detail on the analysis methods for research question 1 is provided in section F.l. 
Section F.1.1 provides information on how student achievement scores were standardized within 
grades and districts; section F.l. 2 describes the analysis model; and section F.l. 3 provides details 
on the exhibits used to present the results for research question 1 in Chapters 5 and 6. 

Overview of Methods for Research Question 2 

One of the primary goals of converting a school to a magnet is to improve the diversity in the 
school. For this study, diversity in a school was measured relative to its district: a school was 
considered to be more diverse when the makeup of its students became more like the makeup of 
its district. Diversity is measured in relative terms because the primary way a school can change 
its composition is to attract students attending other schools in the district — thus, the district’s 
student composition sets the context for any particular school’s improvements in diversity.^*’ In 
this way, for example, a traditional magnet school’s diversity would be viewed as improving 
even if the proportion of students from a minority racial/ethnic background increased (rather than 
decreased, as would be expected), if the district experienced a larger increase. 

Under this approach, when the difference between the proportion of students from minority 
racial/ethnic backgrounds or students who are disadvantaged in a school and in its district is zero 
(no gap), the school population is considered diverse. The larger the difference, the less diverse 
the school. The theory of action, hence, predicts that the difference between the magnet school 
and district proportions will be smaller after conversion than before; that is, the magnet schools 
will become more diverse. 

Describing changes in magnet schools and their districts. The initial step in the analysis 
involved examining whether the diversity in magnet schools increased after conversion by 
determining whether the difference between the magnet school and district proportions of 
students from minority racial/ethnic backgrounds or who were disadvantaged was indeed smaller 
after conversion. 

Investigating the role of magnet conversion. The second step investigated whether magnet 
conversion may have played a role in the observed change in diversity in magnet schools by 
predicting the change in magnet schools had they not converted based on neighborhood schools 


Magnet schools might also attract students from private schools in the area. 
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in the study districts that did not convert.^^ The rationale for this approach is that a reduction in 
the proportion of minority group students or disadvantaged students in traditional magnet 
schools relative to their districts could be caused by factors other than the magnet conversion — 
it could be that all schools in the district were becoming more diverse, including study magnet 
schools, regardless of whether they converted. What happened in neighborhood schools that did 
not convert served as a proxy for what might have happened to the study schools if they had not 
converted.^^ If neighborhood schools that did not convert experienced changes similar to those 
experienced by magnet schools, it would suggest that the magnet school outcomes might be part 
of a larger trend for all neighborhood and magnet schools in the district rather than a result of 
conversion; if the changes in other neighborhood schools were different, then magnet 
conversion is a possible explanation for the difference. This approach is sometimes termed a 
comparative interrupted times series or difference-in-differences analysis. (See Bloom 2003 and 
Somers et al. 2013 for a discussion of the methodology.) Although it is not possible to rule out 
all factors other than conversion with certainty, given the study design, it is possible to provide 
suggestive, but not conclusive, evidence of the role of conversion. 

The methods used to analyze destination schools were similar to the methods described above for 
traditional schools. Destination schools typically start out with a proportion of students from 
minority racial/ethnic backgrounds or from disadvantaged backgrounds lower than their districts, 
whereas traditional schools typically have a proportion higher than their districts. However, 
because the key variable in this analysis was the size of the difference between the school 
proportion and the district proportion, the analysis was similar whether the school proportion was 
greater than or less than the district proportion prior to conversion. In both cases, improved 
diversity entailed a reduction in the size of the difference between the proportion of minority 
students or disadvantaged students in magnet schools and their districts. 

Additional details on the methods for research question 2 are described in section F.2 for racial/ 
ethnic diversity and F.3 for diversity in economic disadvantage. Sections F.2.1 
(racial/ethnic diversity) and F.3.1 (diversity in economic disadvantage) describe the first step of 
the analysis: the comparison of magnet schools to their districts. Sections F.2.2 and F.3. 2 
describe the second step of the analysis: the comparison of magnet schools to neighborhood 
schools that did not convert. 


It is generally preferable to have a comparison group of schools that is well matched to the group of schools 
implementing the practice of interest (here, the conversion to a magnet school) based on their characteristics 
before the practice is implemented. Some common matching methods (e.g., Mahalanobis distance or regression 
based matching approaches, such as propensity score matching) were considered during the design phase of the 
study. However, it was difficult to identify a close match for each magnet school on all relevant school 
characteristics (proportion minority, proportion disadvantaged, and ELA and mathematics achievement), 
especially since in some study districts, there were few schools, limiting the pool of potential matches. In addition, 
a simulation study conducted for the project suggested that the use of all schools in a district might minimize the 
estimation error. For details on the decision to use all noncharter and nonmagnet schools in the district as 
comparisons rather than creating a matched comparison sample, see section F.2.2. 1. For tables comparing the 
study magnet schools and all neighborhood public schools that did not convert on the proportion of students from 
minority backgrounds, the proportion from disadvantaged backgrounds, ELA achievement, and mathematics 
achievement prior to conversion, see Appendix I for traditional magnet schools and Appendix J for destination 
magnet schools. 

The neighborhood public schools used as a comparison did not include magnet schools or charter schools. The 
latter restriction was imposed because none of the magnet schools were charters before they converted. 
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Overview of Methods for Research Question 3 

A primary goal of magnet school conversion is to improve student learning. The theory of action 
suggests two ways that learning might be improved; the adoption of specialized curricula and 
instructional methods; and spillover effects from higher achieving peers. 

Describing changes in magnet schools and their districts. The initial step in examining magnet 
school achievement involved looking at whether the average achievement in ELA and 
mathematics in the magnet schools changed after conversion as anticipated by the theory of 
action. Since the magnet schools were operating within the context of their districts which may 
have had other efforts to improve learning, the change in the achievement in the magnet schools 
was compared to the changes that were occurring in their districts overall. 

This analysis looks at whether students in the years after conversion performed better than the 
students who were in those same grades in the years before conversion. In a typical district, the 
analysis compared the performance of all third, fourth, and fifth grade students in postconversion 
years to third, fourth, and fifth grade students in the preconversion years (different students), in 
order to measure the school’s progress or improvement. (See Exhibit A.3 for the grade ranges 
analyzed for each district.) This approach is similar to how school improvement is measured 
under federal accountability efforts in most states (as required under the Elementary and 
Secondary Education Act).^"^ 

Eor each overall measure, the achievement data were first combined to generate an EEA 
achievement measure for each magnet school for the preconversion period and one for the 
postconversion period. Eor the preconversion measure, the EEA achievement scores of all 
students in grades 3, 4, and 5 in each school were averaged for each preconversion year 
(typically two years). Then, the annual school measures were averaged. Eor the postconversion 
measure, the same process was used to combine the achievement scores (typically over a four- 
year period). A similar approach was used for mathematics achievement in magnet schools and 
for district EEA and mathematics achievement. 

Investigating the role of magnet conversion. As described in Chapter 1, the main population of 
policy interest and the mechanisms for improving learning differ in traditional and destination 
magnet schools. In traditional schools, neighborhood students, who are typically lower 
achieving, are the main focus of policy interest. In contrast, in destination schools, students from 


In traditional magnets, higher aehieving students from outside the neighborhood may boost the aehievement of 
lower aehieving neighborhood students. In destination magnets, higher aehieving neighborhood students may boost 
the aehievement of lower aehieving students from outside the neighborhood. See Chapter 1. 

For example, beeause the students in the third grade in one year were not the same students in the third grade in a 
later year, an inerease in achievement could have been due to actual improvements in learning within the schools, or 
it could have been due to changes in the composition and achievement trajectory of students attending a certain 
grade in a given year. The possibility of the latter is a particular concern in interpreting magnet school achievement 
changes, because the schools specifically aim to change the mix of students attending the schools after conversion, 
which could in turn affect the average achievement in the magnet schools. This issue is addressed in the analysis 
discussed next that compared change in achievement in magnet schools to change in achievement of neighborhood 
schools that did not convert. 
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outside the neighborhood attracted to the magnet school are typically lower achieving and thus 
the main focus of policy interest. 

The key to determining whether any improvement in the achievement of neighborhood students 
in traditional magnet schools might be due to the conversion is comparing changes in 
neighborhood student achievement in traditional magnet schools with what would be predicted 
if the schools had not converted, based on changes in neighborhood student achievement in 
neighborhood schools that did not convert. This strategy, a comparative interrupted time series 
(CITS) analysis, made it possible to investigate whether neighborhood student outcomes in 
magnet schools might have been related to factors affecting a wider set of schools across the 
district, rather than to the conversion. The approach used here was different from the one used to 
measure the achievement change in the magnet schools after conversion (described on prior 
pages) in order to better measure the improvement of individual students. In contrast to looking 
at whether a group of students in the postconversion years performed better than those in the 
same grades in the preconversion years, this analysis compared the annual gains of individual 
students before the conversion with the annual gains of individual students after the conversion. 

For this analysis, the measures analyzed were student-level annual achievement gains: 

■ Individual student gains in ELA achievement from the previous year; and 

■ Individual student gains in mathematics achievement from the previous year. 

Students’ annual achievement gains- that is, their increases (or decreases) from one grade to the 
next — were used to focus the analysis on what students learned while attending a magnet school 
rather what they had learned before. Gains were reported as percentile increases (or decreases) 
from one grade to the next.^^ A positive annual gain in percentile rank indicates that students 
gained ground over the previous year relative to other students in the district; and a negative 
annual gain in percentile rank indicates that students lost ground. Accounting for students’ past 
academic histories by examining gains in their own achievement rather than across groups of 
different students helps rule out changes in the achievement of the school’s neighborhood 
students prior to attending as a reason for any observed change in achievement. For example, as 
noted previously, the students observed in a given grade in one year are not the same students 
observed in that grade in the previous year. Hence, the neighborhood students in a magnet school 
after conversion may be higher achieving than those who attended before conversion because of 
shifting residential patterns rather than as a result of conversion. 

Similarly, it is possible that the social or economic background of the neighborhood students 
after conversion might be different than the background of those who attended before 
conversion. To further focus the analysis on achievement gains as a result of conversion rather 
than changes in student background, the CITS statistical model predicted achievement for 
neighborhood students in magnet schools had their schools not converted based on students with 
similar background characteristics in neighborhood schools that did not convert. The following 


Achievement scores analyzed here were standardized differently than in the analyses described above comparing 
achievement in magnet schools to achievement in their districts. Here students’ achievement scores were expressed 
as percentiles within their district, grade, and year (see Appendix F.4.2.1 for further discussion). 
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characteristics were taken into aeeount: raee/ethnieity, eeonomie disadvantage/^ age, disability 
status, and ELL status. 

Whereas in traditional magnet sehools, the policy interest is the lower performing neighborhood 
students, in destination magnet schools the interest is the lower performing students from outside 
the neighborhood who were attracted to the sehools after they converted. Using these methods to 
investigate the role of eonversion in the outeomes of students from outside the destination 
sehools’ neighborhoods is inappropriate. There are better approaehes for examining the effeets of 
attending a eonverted magnet for these students, ineluding the use of admission lotteries when 
there are more applieants than spaees available. However, the schools were newly eonverted, and 
the number of students who transferred to destination magnet sehools in this study was too small 
for this type of analysis. 

Additional details on the analysis for research question 3 are provided in seetion L.4. Seetion L.4.1 
deseribes the first step of the analysis: the eomparison of aehievement in magnet schools to 
achievement in their distriets. Seetion L.4.2 deseribes the seeond step of the analysis: the 
eomparison of aehievement in magnet sehools to aehievement in neighborhood sehools in their 
distriets that did not eonvert. 

Additional Information About Methods 

After deseribing the methods used to answer eaeh of the researeh questions. Appendix L 
eoneludes with two supplemental seetions. Seetion L.5 provides a diseussion of potential bias in 
the analyses of ehange in diversity in magnet sehools relative to change in neighborhood schools 
that did not convert. Seetion L.6 provides notes on supplementary analyses eondueted to examine 
potential explanations for the ELA aehievement results presented in Chapter 5. 

F.1. Analysis Methods for Research Question 1 

The analyses for researeh question 1 in Chapters 5 and 6 were eondueted using annual school-level 
measures for eaeh study magnet sehool. Student-level data were aggregated for eaeh year for eaeh 
study magnet sehool to generate sehool-level measures for two student groups: students from 
outside the neighborhood and neighborhood students. To simplify analysis, for these caleulations 
students were assigned to the elementary sehool in whieh they were enrolled for the greatest portion 
of the sehool year (i.e., the dosages and test-to-test years deseribed in Appendix B were not used). 

The following annual measures were ealeulated separately for eaeh study magnet sehool, for 
students from outside the neighborhood and neighborhood students: 

■ Count of students 

■ Proportion of students from minority raeial/ethnie baekgrounds 


In three districts, data on economic disadvantage were not available. In one of those districts, information on the 
level of parent education was available and used as a substitute measure of socioeconomic status in this analysis. 

Students who did not have a known residency status were not included in the analysis reported for research 
question 1. 
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■ Proportion of students from economically disadvantaged backgrounds 

■ Average ELA achievement 

■ Average mathematics achievement 

These measures were then used to create additional school-level measures for each year. First, 
the annual counts of students were used to calculate the annual share of students in each school 
who were from outside the neighborhood. The share was calculated by dividing the count of 
students from outside the neighborhood by the sum of the count of students from outside of the 
neighborhood and the count of neighborhood students. Second, the following differences 
between students from outside the neighborhood and neighborhood students were calculated: 

■ Difference in the proportion of students from minority racial/ethnic backgrounds 

■ Difference in the proportion of students from economically disadvantaged backgrounds 

■ Difference in average ELA achievement 

■ Difference in average mathematics achievement 

F.1.1. Standardized Achievement Scores 

The analyses of achievement for research question 1 were conducted using test scores 
standardized within district within grade, with respect to the district’s base year (first year of data 
for the district). Each student’s test score was converted to a z-score by subtracting the district 
base-year mean for the student’s grade level and dividing by the district base-year standard 
deviation for that grade level. 

F.1.2. The Analysis Model 

A two-factor ANOVA model was used to conduct the analyses reported for research question 1 . 
The model was estimated using each of the measures described previously as the dependent 
variable and “period (pre-post)” and “school” as the two factors. Elsing period and school as 
factors allowed the analysis to pool data across years (e.g., the observations for two 
preconversion years and four postconversion years) and across schools (e.g., across 17 traditional 
magnet schools for analysis in Chapter 5 and across four destination schools in Chapter 6). 
Specifically, the following model was estimated: 

Vs.p.t ~ P T oc^ -f (Bp -f (o[(B) 5_(- -f ^s,p,t (F-1) 


where 

■ ys,p,t = an outcome measure for school 5, in period p, and year t 

■ s = index for schools 

■ p = “pre” for observations in the preconversion period (typically two years) or “posf ’ for 
observations in postconversion years (typically four years) 

■ t= index for the year the data were observed 

■ p, = the overall mean 
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■ the main effect for MSAP school s 

■ Pp = the main effect for period p, where p = “pre” for observations in the preconversion 
period or “post” for observations in postconversion years 

The estimated coefficients from this model were used to provide averages across study magnet 
schools for the preconversion period (p, + Ppre) and the postconversion period (p + Ppost)^ as 
well as the average change from the preconversion to the postconversion period (Ppost “ Ppre) 
for each dependent variable. 

First, equation F.l was estimated using the annual share of students who were from outside the 
neighborhood as the dependent variable and tested whether the postconversion average share 
among magnet schools differed from the preconversion share {Ho: Ppost ~ Ppre = 0)- 

Next, equation F.l was estimated three times to determine whether there was a change in the 
proportion of students from outside the neighborhood who were from minority racial/ethnic 
backgrounds, a change in the proportion for students from the neighborhood, and a change in the 
difference between the two.^^ 

■ To test whether the postconversion average proportion of raciaFethnic minority students 
for students from outside the neighborhood differed from the preconversion average 
{Ho: Ppost “ Ppre — 0)5 equation F.l was estimated using the proportion of students from 
minority raciaFethnic backgrounds outside the neighborhood as the dependent variable. 

■ To test whether there was a change for neighborhood students {Ho: Ppost “ Ppre = 
equation F.l was estimated again, using the proportion of raciaFethnic minority students 
for neighborhood students as the dependent variable. 

■ Finally, to test whether there was a preconversion difference between students from 
outside and inside the neighborhood {Ho: p. + Ppre = 0), a postconversion difference 
{Ho: p + Ppost = 0)^ and a change in the difference {Ho: Ppost “ Ppre = 0)^ the model 
was estimated a third time, using the difference in the proportion of racial/ethnic minority 
students (the proportion for students from outside the neighborhood minus the proportion 
for neighborhood students) as the dependent variable. 

The same approach was used to analyze the proportion of students from economically 
disadvantaged backgrounds, average ELA achievement, and average mathematics achievement 
by estimating equation F. 1 for students from outside the neighborhood, for neighborhood 
students, and for the difference between them. 


Estimating equation F. 1 three times produeed the results of the five different signifieanee tests that are reported in 
Exhibit 5.2 (repeated as Exhibit F.2): the ehange in the proportion of students from outside the neighborhood from 
minority baekgrounds, the ehange in the proportion of neighborhood students from minority baekgrounds, the 
preeonversion differenee between students from outside the neighborhood and neighborhood students, the 
posteonversion differenee, and the ehange in the differenee. 
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F.1.3. Exhibits 


The exhibits in Chapters 5 and 6 under research question 1 present estimates and hypothesis 
tests based on equation F.l and the analysis described in section F.1.2. Three examples from 
Chapter 5 illustrate how the exhibits were produced; 

■ The share of students who were from outside the neighborhood (Exhibit 5.1) 

■ The proportion of students from minority raciaFethnic backgrounds (Exhibit 5.2) 

■ The average ELA achievement (Exhibit 5.3) 

Exhibit 5.1, reproduced here as Exhibit E.l, displays the share of students who were from outside 
the neighborhood. These results were generated by estimating equation E. 1 using the annual 
share of students who were from outside the neighborhood as the dependent variable: the average 
share of students who were from outside the neighborhood was 21.0 percent before and 
26.8 percent after conversion. The pre-post change in the average share ((Bpost — (Bp^e) is 
provided on the display (5.8 percentage points) between the preconversion and postconversion 
numbers. The statistical significance of the change (Ho: (Bpost ~ Ppre = 0) is indicated with an 
asterisk (*) if the change is statistically different than zero (p < .05) and no symbol if it is not 
different. In Exhibit 5.1, the change is significant. 

Exhibit F.1. Reprint of Exhibit 5.1. Share of Students From Outside the Neighborhood in 
Traditional Magnet Schools (Average Across Schools) 
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Exhibit 5.2, reproduced here as Exhibit E.2, presents the results for the proportion of students 
from minority racial/ethnic backgrounds in traditional magnet schools. Eirst, based on 
equation E.l using the annual proportion of racial/ethnic minority students for students from 
outside the neighborhood as the dependent variable, the average proportion of racial/ethnic 
minority students was 84.4 percent both before and after conversion. As in Exhibit 5.1, the 
pre-post change ((Bpost “ Ppre) significance test {Ho: Ppost “ Ppre = 0) are provided on the 
exhibit between the preconversion and postconversion numbers. The absence of an asterisk 
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(i.e., there is no *) next to the change (which rounds to zero) indicates that the postconversion 
average proportion was not statistically different from the preconversion average proportion 
ip < .05). 

Second, based on equation F.l and using the annual proportion of racial/ethnic minority students 
for neighborhood students as the dependent variable, the average minority proportion was 
83.0 percent before and 84.0 percent after conversion. The absence of an asterisk (i.e., there is 
no *) next to the change (1.0 percentage points) indicates that this difference was not statistically 
significant {Ho: (Bpo^t - (Bp^e ^0;p< .05). 

Exhibit F.2. Reprint of Exhibit 5.2. Characteristics of Neighborhood Students and 
Students From Outside the Neighborhood in the Traditional Magnet 
Schools (Average Across Schools) - Proportion From Minority 
Racial/Ethnic Backgrounds 
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Third, based on equation F.l and using the annual difference in the proportion of racial/ethnic 
minority students between students from outside the neighborhood and neighborhood students 
as the dependent variable, the average difference in minority proportion was 1 .4 percentage 
points before and 0.5 percentage points after conversion. The statistical significance of the 
difference in proportion between students from outside the neighborhood and neighborhood 
students, pre-conversion {Ho: p. + (Bpj-e = 0) and post-conversion {Ho: p + (Bpost — 0)> 
respectively, is reported in the display using the following notation: 

■ An asterisk (*) next to the average preconversion difference or next to the average 
postconversion difference indicates that the average preconversion or postconversion 
difference was statistically different than zero {p < .05). 

■ No symbol next to the preconversion difference or the postconversion difference 
indicates that the average preconversion or postconversion difference was not statistically 
different than zero. 
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In Exhibit 5.2, the absence of an asterisk (i.e., there is no *) next to each average indicates that 
neither the difference before nor after conversion was statistically significant. 

In addition, the change in the average difference from the preconversion to the postconversion 
period is reported in the text box in the display: -1.0 percentage point. An asterisk (*) is used to 
indicate statistical significance (Ho: Ppost “ Ppre — 0)- The absence of an asterisk (i.e., there is 
no *) beside the change in the difference indicates that the postconversion average difference 
was not statistically different from the preconversion average difference. 

The numbers in Exhibit 5.3, reproduced as Exhibit F.3, present results for the average EEA 
achievement in traditional magnet schools for students from outside the neighborhood and 
neighborhood students. The displays for achievement results for research question 1 were 
constructed as described for the proportion of racial/ethnic minority students in reference to 
Exhibit E.2. Averages and hypothesis tests were generated by estimating equation E.l using 
three different dependent variables: the average achievement for students from outside the 
neighborhood, the average achievement for neighborhood students, and the difference in average 
achievement between the two. However, the achievement displays require additional explanation 
because the achievement results were presented in terms of percentiles rather than the z-scores 
used for analysis. 

Exhibit F.3. Reprint of Exhibit 5.3. Achievement of Neighborhood Students and 

Students From Outside the Neighborhood in Traditional Magnet Schools 
(Average Across Schools) - ELA 
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In the report, the z-score results were transformed into pereentiles for display to put the results in 
a policy-relevant metric. The transformation was conducted using the standard normal 
cumulative distribution function (CDF) to assign each z-score result a percentile rank: 

Percentile score = O(z-score) (F.2) 

Because it was assumed that z-scores were normally distributed, each z-score should 
correspond to a percentile in the standard normal distribution that can be found using the 
standard normal CDF. For students from outside the neighborhood in traditional magnet 
schools, the preconversion average ELA z-score was -0.37, and the postconversion average 
z-score was -0.15; these achievement levels correspond to percentile ranks of 35.6 and 44.0, 
respectively (see Exhibit E.3). Eor neighborhood students, the preconversion average EEA 
z-score was -0.33, and the postconversion average z-score was -0.14; these achievement levels 
correspond to percentile ranks of 37.1 and 44.4, respectively (see Exhibit E.3). 

Exhibit 5.3 also displays differences in average achievement between students from outside the 
neighborhood and neighborhood students. The average EEA z-score difference between students 
from outside the neighborhood and neighborhood students was -0.04 in the preconversion period 
and -0.01 in the postconversion period. However, rather than transforming these z-scores to 
percentiles, the percentiles for display in Exhibit 5.3 were calculated as the difference between 
the average percentile for students from outside the neighborhood and the average percentile for 
neighborhood students.^® Eor the preconversion period, the difference was -1.5 percentile points 
(= 35.6 - 37.1), and for the postconversion period, the difference was -0.4 percentile points 
(= 44.0 - 44.4). The change in the difference, +1.1 percentile points (as reported in the text box), 
was calculated by subtracting the preconversion average difference in percentiles from the 
postconversion difference: 1.1 = -0.4 - (-1.5). 

Although the results in Exhibit 5.3 are reported in percentiles, the statistical tests were conducted 
using the original z-scores. The results of the hypothesis test are reported in Exhibit 5.3, using 
asterisks (*) as described for Exhibit 5.2: 

■ Pre-post change for students from outside the neighborhood. The asterisk (*) that 
appears next to the change in average EEA percentile rank for students from outside the 
neighborhood (8.4 percentile points) indicates that the difference between the 
postconversion percentile rank, 44.0, and the preconversion percentile rank, 35.6, was 
statistically significant. 

■ Pre-post change for students for neighhorhood students. The asterisk (*) that appears 
next to the change in average EEA percentile rank for neighborhood students 


*** As described in the text, the difference in z-scores was converted to a difference in percentiles by converting each 
of the two terms in the difference separately and then subtracting. This approach was used because the percentile 
difference corresponding to a given z-score difference differs depending on the base from which the z-score 
difference is measured. For example, consider a z-score difference of +0.1. If this difference is computed from a 
base of 0.0 to a value of 0. 1, the change in percentile would be from the 50th to the 54th percentile, a difference of 
4 percentile points. However, if the difference is computed from a base of -1.0 to a value of -0.9, the change in 
percentile would be from the 16th to the 18.4th percentiles, a difference of 2.4 percentile points. 
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(7.3 percentile points) indicates that the difference between the postconversion percentile 
rank, 44.4, and the preconversion percentile rank, 37.1, was statistically significant. 

■ Preconversion difference between students from outside the neighborhood and 
neighborhood students. The absence of an asterisk (i.e., there is no *) next to the 
preconversion average difference, -1.5, indicates that before conversion the average ELA 
percentile rank for students from outside the neighborhood was not statistically different 
than for neighborhood students. 

■ Postconversion difference between students from outside the neighborhood and 
neighborhood students. The absence of an asterisk (i.e., there is no *) next to the 
postconversion average difference, -0.4, indicates that after conversion the average ELA 
percentile rank for students from outside the neighborhood was not statistically different 
than for neighborhood students. 

■ Pre-post change in the difference between students from outside the neighborhood 
and neighborhood students. The absence of an asterisk (i.e., there is no *) beside the 
change in the difference of 1.1 points shown in the text box indicates that the 
postconversion average difference was not statistically different from the preconversion 
average difference. 

F.2. Analysis Methods for Research Question 2 - Concentration of 
Students From Minority Racial/Ethnic Backgrounds 

Research question 2 addresses the concentration of students from minority racial/ethnic groups 
and the concentration of students from economically disadvantaged backgrounds in schools 
relative to their districts. This section discusses methods for the analysis of minority 
concentration; the next section, E.3, discusses methods for the concentration of economically 
disadvantaged students. The analyses for this research question were conducted in two ways. 

The first set of analyses compared the proportion in magnet schools to the proportion in their 
districts; a description of these analyses appears in section E.2.1. The second set of analyses 
compared the difference between magnet schools and their districts to the difference between 
neighborhood schools that did not convert and their districts; a description of these analyses 
appears in section E.2.2. 

F.2.1. Changes in Magnet Schools and Their Districts 

The first step in the analysis of minority concentration for research question 2 used equation E.l 
to compare study magnet schools to their districts overall. This analysis was similar to the 
comparison of students from outside the neighborhood to neighborhood students in research 
question 1. Eor each magnet school, student-level data for all students in the school were 
aggregated within the year to the school level to generate the annual measures. Eor each district, 
student-level data for all students in all public schools in the district (including study magnet 
schools) were aggregated within year to the district level. In some cases, multiple study magnet 
schools were located in the same district. When an average for the districts in which the study 
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magnet schools were located was obtained, districts that had multiple study magnet schools were 
included multiple times, once for each study school in the district.^' 

The following annual measures were calculated for each study magnet school: 

■ Proportion of students from minority racial/ethnic backgrounds in the magnet school 

■ Proportion of students from minority racial/ethnic backgrounds in the magnet school’s 
district 

These annual measures were then used to calculate the following annual difference between each 
magnet school and its district: 

■ Difference in proportion of students from minority racial/ethnic backgrounds 

F.2.1.1. The Analysis Model. As for research question 1, equation F.l was used to analyze 
minority concentration for research question 2. To test whether the postconversion average 
proportion of students from minority racial/ethnic backgrounds for magnet schools differed from 
the preconversion average, equation F.l was estimated once using the minority proportion for 
magnet schools as the dependent variable. Then equation F.l was estimated again, based on the 
proportion of raciakethnic minority students for the districts in which the magnet schools were 
located as the dependent variable. Finally, to test whether there was a difference in the 
proportion of raciakethnic minority students between magnet schools and their districts in the 
preconversion period or in the postconversion period, and whether there was a change in the 
difference between the pre- and postconversion periods, the model was estimated a third time, 
using the difference in minority proportion, the proportion in each magnet school minus the 
proportion in its district, as the dependent variable. 

F.2.1.2. Exhibits. Exhibit 5.4, reproduced here as Exhibit E.4, presents results for the proportion 
of students from minority raciakethnic backgrounds in traditional magnet schools and their 
districts. The presentation of results is similar to the presentation of results for students from 
outside the neighborhood and neighborhood students as described in section E.1.2. Eirst, based 
on equation E.l using the annual proportion of raciakethnic minority students for students in 
study magnet schools as the dependent variable, the average minority proportion was 84.5 
percent before and 84.9 percent after conversion. As in Exhibit 5.2, the pre -post change in the 
average proportion (0.4 percentage points) is provided on this display between the preconversion 
and postconversion numbers. The absence of an asterisk (i.e., there is no *) next to the change 
indicates that it was not statistically different than zero. 

Second, based on equation E.l using the annual proportion of raciakethnic minority students in 
the districts of traditional magnet schools as the dependent variable, the average minority 
proportion was 64.1 percent before and 66.3 percent after conversion. The asterisk (*) next to 


As an example, Distriet D eontained three traditional study magnet sehools. Henee, when the average 
eoneentration of students from a minority raeial/ethnie baekground aeross study distriets for traditional magnet 
sehools was obtained, the Distriet D eoneentration entered the analysis three times, onee for eaeh traditional magnet 
sehool. 
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pre-post change in the average proportion (2.3 percentage points) indicates that the change was 
statistically different than zero. 

Third, based on equation F.l using the annual difference between the proportion of racial/ethnic 
minority students in magnet schools and their districts as the dependent variable, the average 
difference in minority proportion was 20.4 percentage points before and 18.5 percentage points 
after conversion. As in Exhibit 5.2, asterisks report the results of hypothesis tests: 

■ The asterisk (*) next to the preconversion average difference, 20.4, indicates that before 
conversion the average proportion of racial/ethnic minority students in magnet schools 
was statistically different from the proportion in their districts. 

■ The asterisk (*) next to the postconversion average difference, 18.5, indicates that after 
conversion the average proportion of racial/ethnic minority students in magnet schools 
was statistically different from the proportion in their districts. 

■ The asterisk (*) next to the change in the difference, -1.9, indicates that the 
postconversion average difference was statistically different from the preconversion 
average difference. 

Exhibit F.4. Reprint of Exhibit 5.4. Concentration of Racial/Ethnic Minority Students in 
Traditional Magnet Schools (Average Across Schools) 
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F.2.2. Investigating the Role of Magnet Conversion 

This section describes the methods for the second step in the analyses of the concentration of 
students from minority racial/ethnic backgrounds for research question 2 in Chapters 5 and 6. As 
discussed in Chapter 2, the primary approach for determining whether magnet conversion played 
a role in the observed outcomes was to use a CITS analysis to compare changes in study magnet 
schools to what would be predicted had they not converted. The prediction is based on changes 
in neighborhood public schools in the same district that did not convert to magnet schools. 
Section F.2.2. 1 discusses the construction of the comparison group used in the analysis, and 


F-15 


What Happens When Schools Become Magnet Schools? — Appendices 


section F.2.2.2 provides details of the analysis. Section F.2.2.3 explains the meta-analysis used to 
aggregate the findings across districts, and section F.2.2.4 discusses the displays. 

F.2.2.1. Construction of the Comparison Group for Predictions. The comparison schools used in 
the analyses for research question 3 were defined as public elementary schools in the district that 
were never charter or magnet schools and were in continuous operation throughout the years of 
the study. The study magnet schools were all noncharter, nonmagnet schools prior to conversion, 
and it was believed that the trajectory for these schools would serve as a reasonable 
representation of what might have happened in magnet schools had they not converted. 

The decision to use all regular (i.e., noncharter, nonmagnet) neighborhood schools in the district 
that did not convert as a comparison group rather than an alternative approach was made based 
on the realities of the data available for this study and results from a simulation study that 
investigated alternatives (Betts et al. 2009). The original study design proposed to use an 
individually matched comparison school from within the district for each study school as a 
control group. However, the identification of close matches proved to be difficult for some 
magnet schools located in districts with a small total number of schools. The small number of 
schools in some districts also made the propensity score matching method for creating a control 
group infeasible. 

The simulation study, Betts et al. 2009, was conducted as a companion to this study to 
investigate various methods for establishing a control group. Four methods were compared; 
using regressions to individually match study schools to a comparison school, a synthetic control 
group method (Abadie and Gardeazabal 2003), propensity score matching, and using all other 
neighborhood public schools. The simulation study was based on school-level data for schools in 
24 districts located in three states. Data on school-level ELA and mathematics test scores were 
obtained from the National Longitudinal School Level State Assessment Score Database 
(NLSLSASD), and data on demographics were obtained from the National Center for Education 
Statistics (NCES) Common Core of Data for school years 2001-02 through 2006-07. The study 
used a “falsification” approach to study each method. Within each of the 24 districts, a school 
was randomly selected to be designated as having converted to a magnet school, with probability 
related to the school’s prior achievement, to reflect the idea that lower performing schools are 
more likely to be selected for magnet conversion. A comparison school or group was then 
selected according to each of the four methods; and finally a regression model was used to 
estimate the hypothetical relationship between magnet status and achievement. Since in the 
simulation there was no true effect of magnet status, the models should not find a significant 
association between conversion and achievement. This approach was repeated 1,000 times for 
each method, drawing different pseudo-magnet schools in each district, to determine which 
method most accurately rejected the hypothesis that the (false) conversion was associated with a 
change in achievement. Using all other neighborhood public elementary schools in the district as 
a comparison group was found to minimize the estimation error. 

F.2.2.2. Analysis Model. To assess whether there was an association between magnet conversion 
and a change in the proportion of students from minority racial/ethnic backgrounds, the analysis 
was based on the absolute value of the difference (AVD) in the proportion of racial/ethnic 
minority students (AVDm), expressed as follows: 
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AVDm = I minority proportion in the school - minority proportion in its district | 

The absolute value of these differences was calculated because, as noted in Chapter 2, even 
within traditional or destination magnet schools, there were some schools with proportions 
greater than their district and some less than their district. In addition, AVD was calculated for 
each of the neighborhood schools that did not convert, some of which had proportions greater 
than their districts and some less. Taking the absolute value of the difference put all of the 
schools onto the same metric of diversity: size of the distance from the district proportion. 

The following CITS model for AVDm was estimated: 

AVDMst = iLi V-s+T^iFzt + + T^^Fst + TTsFgt + eTREAT^t + Sst (F.3) 

where 


■ s = index for schools 

■ t = index for the year the data were observed 

■ |Ui = an indicator variable for school s 

■ Fit ... F6t = year indicator variables for Years 2 through 6, respectively 

■ 711 ... 715 = estimated year effects 

■ TREATii = 1 if school s is an MSAP school in year t when the program was active; 0 
otherwise 

The coefficient on TREAT, 0, represents the association between magnet conversion and the 
change in AVD between the postconversion and preconversion period for magnet schools. The 
error term — — represents variation in AVD within each school across years. In estimating the 
model, clustering at the school level was allowed to account for time-varying correlations within 
the set of observations for a given school that are not captured by the fixed effects. 

The models were estimated separately for each district, to allow for complete flexibility in the 
coefficients across districts. 

Equation E.3 produced a single estimate 0 for each district, representing the association between 
conversion to a magnet school and change in the absolute value of the difference between the 
school and the district across all study magnet schools in the district. These estimates were used 
in the meta-analysis reported in research question 2 and described in section E.2.2.3. 


The CITS model shown in F.2 also included a “trend” variable that was set equal to zero for each year in the 
preconversion period and -1.5, -0.5, 0.5, and 1.5 for postconversion Years 1, 2, 3, and 4, respectively. The analysis 
included interaction terms between the trend variables and the TREAT variables to examine whether the changes 
associated with magnet school conversion increased or declined during the postconversion period. For simplicity, 
the trend variable is not shown in equation F.3, but the results for the trend are reported in Exhibit 1.19 for traditional 
magnets and Exhibit J.13 for destination magnets. The coding of the trend variable was designed so that its inclusion 
in the model would not affect the estimate for the TREAT variable. 
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F.2.2.3. Meta-Analysis of District-Level Results. For aggregating results aeross districts, the 
study used a meta-analysis of the district-specific estimates to produce overall study average 
estimates. The me ta- analysis treats each district model as a separate study and obtains an overall 
coefficient based on coefficients from the separate studies. 

The meta-analysis was based on a random-effects model, which assumes that the separate district 
estimates represent a population of districts with magnet programs that may vary in their effects, 
and the focus is on estimating the population average association between magnet conversion 
and change in the absolute value of the difference in the proportion of students from minority 
racial/ethnic or economically disadvantaged backgrounds. The assumption of varying effects was 
based on a recently conducted meta-analysis of the literature on charter schools. In their analysis, 
Betts and Tang (2011) reasoned that “[gjiven that charter schools are afforded considerable 
freedom to experiment, and that the regulatory framework for charter schools varies across 
states, and surely across individual districts as well, it would seem untenable to make the 
alternative assumption that there is a single fixed impact of charter schools on achievement.” 

It was reasoned that these same points apply to magnet schools.*^ The random effects meta- 
analysis approach produces an overall estimate and standard error for the population treatment 
coefficient, and in addition, it provides statistics on the extent to which variation in these 
coefficients across districts is real rather than due to sampling variation. 


In a random effects meta-analysis, the population estimate is based on a weighted average of the 
effect sizes across studies. If 0j is the effect size for the zth of k studies and Wi is the weight for 
each study, the overall estimated effect size Mis as follows: 


M = 




(F.4) 


For example, for the analysis of the absolute value of the difference between the magnet school 
and its district in proportion of racial/ethnic minority students (AVDm), the study calculated the 
overall effect size, M across all the district coefficients, 0j, estimated by equation F.4. The 
weight for each district (i.e., study), Wj, was computed as the inverse of the sum of the within- 
study variance Vq. (based on the standard error) and an estimate of the true between-study 
variance, T^\ 


Wi = 


Vq.+T^ 


(F.5) 


The between-study variance estimate, T^, is based on a method-of-moments estimate of the 
variance of true effect sizes. If is large relative to the average within-study variance estimate, 
the approach produces roughly equal weights across studies, whereas if is relatively small, the 
approach produces unequal weights, with heavier weight given to studies with the lowest 
sampling variance. 


For a review of the random-effeets approaeh to meta-analysis and measures of heterogeneity, see Borenstein et al. 
(2009), Chapters 12 through 16. 
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This study reports the fi statistie introdueed by Higgins et al. (2003), which provides an 
estimate of the percentage of the variation in effect sizes that reflects true underlying variation 
(as opposed to sampling variation). 

The standard meta-analytical approach assumes that the studies are drawn from independent 
samples. Some of the study districts have two or three MSAP-funded (study) magnet schools. 
Even though the analysis controlled for school and residency effects and district time trends as 
shown in equation F.3, there may be a nonzero covariance in effect for estimates across study 
magnet schools operating in the same district. If the analysis were to treat each study magnet 
school estimate as an independent study, then the analysis would likely overstate the precision 
with which it estimated our overall effect. Thus, in the case of districts with more than one study 
magnet school, the analysis used the districtwide estimate described previously that pools the 
magnet schools within a district together. This approach ensures that results do not overstate the 
number of studies and thus overstate the precision of the overall meta-analytic estimate. 

F.2.2.3.1 Power Analysis for Racial/Ethnic Diversity CITS Estimates. Exhibit F.5 shows the post-hoc 
minimum detectable effect (MDE) for the estimates of the “change associated with magnet 
conversion” reported for traditional magnet schools in Chapter 5 and destination magnet schools in 
Chapter 6. The post-hoc MDE is a measure of the MDE actually realized by the study design and 
sample. The estimated MDE is derived by multiplying the empirical standard error of each 
coefficient from the models described previously (equation F.4) by a factor of 2.8.^"^ This calculation 
is based on a two-sided t-test of the coefficient at the 5 percent level, setting power to 0.8. 


Exhibit F.5. MDE for Estimates of the Role of Magnet Conversion in the Concentration 
of Racial/Ethnic Minority Students in Traditional Magnet Schools (Alpha = 
0.05, Power of 0.8) 


Subject 

Standard Error 

MDE 

Traditional magnet schools: Minority proportion 
(AVDM) (%) 

0.740 

2.07 

Destination magnet schools: Minority proportion 
(AVDM) (%) 

1.880 

5.26 


SOURCE: District administrative data. 


F.2.2.4 Exhibits. The CITS estimates derived from the meta-analysis were illustrated with a 
three-point graph. In this section, the analysis of AVDm between traditional magnet schools and 
their districts is used as an example to illustrate how the graphs were constructed. 


The factor of 2.8 is the sum of taipha/2 and ti - beta, where alpha is the intended significance level for the test (0.05), and 
1 - beta is the intended power (0.80). For large degrees of freedom, taipha/2 = 1-96 and ti -beta = 0.84. See Bloom (2006). 
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Exhibit F.6. Reprint of Exhibit 5.5. The Role of Magnet Conversion in the Concentration 
of Racial/Ethnic Minority Students in Traditional Magnet Schools (Average 
Across Schools) 
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Exhibit 5.5, reproduced as Exhibit E.6, shows the CITS estimate (Min equation F.4) for the 
association between magnet conversion and the absolute value of the difference in proportion of 
students from minority racial/ethnic backgrounds between traditional magnet schools and their 
districts (AVDm); -1.9 percentage points. The CITS result can be interpreted as the difference 
between the actual AVDm for traditional magnet school in the postconversion period and the 
predicted AVDm in the postconversion period had the schools not converted. In Exhibit E.6, we 
present the CITS result using this interpretation as the rationale for a three-point display. The 
display shows the level of AVDm in magnet schools as well as the level of AVDm that our 
analysis predicted for magnet schools in the postconversion period had they not converted. The 
remainder of this section describes the derivation of the numbers displayed in Exhibit F.6: two 
of the three points were derived from a descriptive analysis, while the remaining point was 
calculated from the other two numbers in the display and the CITS estimate.*^ Exhibit F.7 
summarizes the derivation of each point. 


In a typical CITS analysis, a display similar to Exhibit F.6 could be constructed directly from the regression 
coefficients, using regression-adjusted preconversion and postconversion averages for magnet and comparison 
schools. This study, however, has an additional complication because the CITS estimate was generated using meta- 
analysis, by aggregating separate CITS estimates based on separate regression for each district (see section F.2.2.3). 
To simplify construction of the display, preconversion and postconversion averages generated by the ANOVA 
model (equation F. 1) were used as the basis for the display, rather than aggregated regression-adjusted averages 
across districts. 
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Exhibit F.7. Derivation of Numbers in Exhibit 5.5 


Description 

Number 

Derivation 

CITS estimate: Change associated with 
conversion 

-1.9 

M estimated from equation F.4 

Study magnet schools: Preconversion 

21.7 

Equation F.l with AVD as dependent variable 

Study magnet schools: Postconversion 

19.8 

Equation F.l with AVD as dependent variable 

Study magnet schools: Postconversion — 
without conversion (predicted) 

21.7 

(Study magnet postconversion) - (CITS estimate) 


The average AVDm for study magnet sehools in the preconversion period, 21.7 percentage 
points, was obtained by estimating equation F.l (the ANOVA model used to generate results for 
the first part of research question 2) using school-level AVDm as the dependent variable. The 
study used this approach because the meta-analysis that aggregated results across districts 
provided only the CITS estimate and not the other data points in Exhibit F.6. Equation E.l 
provides the appropriate preconversion and postconversion averages for magnet schools, 
employing the same method used to produce the averages presented for the comparison of 
magnet schools to their districts in section E.2.1.*® The same approach was used to estimate the 
average AVDm for study magnet schools in the postconversion period, 19.8. 

The predicted AVDm for study magnet schools in the postconversion period had they not 
converted, 21.7 percentage points, was calculated as the average postconversion AVDm for study 
magnet schools minus the CITS estimate (21.7 = 19.8 - [-1.9]). Hence, the CITS estimate 
appears in the graph as the difference between the observed postconversion average for study 
magnet schools and the predicted postconversion average had they not converted. 

F.3. Analysis Methods for Research Question 2 - Concentration of 
Students From Economically Disadvantaged Backgrounds 

Research question 2 also addresses the concentration of students from economically 
disadvantaged backgrounds. As with the analyses of the concentration of racial/ethnic minority 
students, the analyses for disadvantaged students were conducted in two steps. The first step 
compared the proportion in magnet schools to the proportion in their districts; a description of 
these analyses appears in section F.3.1. The second step compared the difference between 
magnet schools and their districts to the difference between neighborhood schools that did not 
convert and their districts; a description of these analyses appears in section F.3.2. 


Although the method is the same, the dependent variable used for the comparison of magnet schools to their 
districts is different than the dependent variable used here. For comparisons of magnet schools to their districts, the 
exhibits report the average difference between the magnet school concentration and the district concentration, 
whereas for predicting what would have happened to magnet schools had they not converted based on neighborhood 
schools that did not convert (the CITS analysis), exhibits report the average of the size (absolute value) of the 
difference between magnet schools and their districts. 
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F.3.1. Comparing Changes in Magnet Schoois and Their Districts 

The first step in the analysis of the eoneentration of eoonomieally disadvantaged students for 
researeh question 2 was the same as the analysis of minority eoneentration (as deseribed in 
section F.2.1) but used the proportion of students from economically disadvantaged backgrounds 
instead of the proportion from minority racial/ethnic backgrounds. 

Similar to the analysis in section F.2.1, the following annual measures were calculated for each 
study magnet school; 

■ Proportion of students from economically disadvantaged backgrounds in the magnet 
school 

■ Proportion of students from economically disadvantaged backgrounds in the magnet 
school’s district 

These annual measures were then used to calculate the following annual difference between each 
magnet school and its district: 

■ Difference in proportion of students from economically disadvantaged backgrounds 

The same analysis model, as described in section F.2.1.1, and displays, as described in section 
F.2.1. 2, were used with these measures. 

F.3.2. Investigating the Roie of Magnet Conversion 

The second step in the analyses of disadvantaged concentration for research question 2 in 
Chapters 5 and 6 was also conducted using a CITS analysis, as was done for the analyses of 
minority concentration, as described in section F.2.2. Here, however, the analysis used the 
proportion of students from economically disadvantaged backgrounds instead of the proportion 
from minority racial/ethnic backgrounds. The same comparison group, as described in section 
F.2.2. 1, was used. In addition, the same analysis model (as described in section F.2.2. 2), meta- 
analysis (as described in section F.2.2. 3), and exhibits (as described in section F.2.2. 4) were 
used, but with the absolute value of the difference between the school proportion and district 
proportion of students from economically disadvantaged backgrounds, AVDd, as the key 
measure: 

AVDd = I free or reduced-price lunch proportion in the school - free or reduced-price 
lunch proportion in its district | 

As done for the analyses of minority concentration, the CITS analysis of AVDd was conducted 
using equation F.2. 

F.3.2. 1 Power Analysis for Socioeconomic Diversity CITS Estimates. Exhibit F.8 shows the 
post-hoc minimum detectable effect (MDE) for the estimates of change associated with magnet 
conversion reported for traditional magnet schools in Chapter 5 and destination magnet schools 
in Chapter 6. The post-hoc MDE was calculated as described in section E.2.2.3.1. 
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Exhibit F.8. MDE for Estimates of the Role of Magnet Conversion in the Concentration 
of Economically Disadvantaged Students in Traditional Magnet Schools 
(Alpha = 0.05, Power of 0.8) 


Subject 

Standard Error 

MDE 

Traditional magnet schools: Disadvantaged proportion 
(AVDD) (%) 

1.608 

4.50 

Destination magnet schools: Disadvantaged proportion 
(AVDD) (%) 

3.092 

8.66 


SOURCE: District administrative data. 


F.4. Analysis Methods for Research Question 3 

Research question 3 addresses student achievement in ELA and mathematics. The analyses for 
this research question were conducted in two ways. The first set of analyses compared the 
achievement in magnet schools to the achievement in their districts; a description of these 
analyses appears in section F.4.1. The second set of analyses compared the annual gain in 
achievement for neighborhood students in magnet schools to the annual gain in achievement for 
neighborhood students in neighborhood schools that did not convert; a description of these 
analyses appears in section F.4.2. 

F.4.1. Changes in Magnet Schools and Their Districts 

The analyses comparing magnet schools to their districts for research question 3 in Chapters 5 
and 6 were conducted using an approach similar to that used for analyses for research questions 2 
and 3. Tike those for research question 2, these analyses used annual measures for study magnet 
schools and their districts. Whereas the exhibits for research questions 2 and 3 present results for 
concentrations of racial/ethnic minority students and economically disadvantaged students, 
respectively, the exhibits for research question 3 present results for average EFA and 
mathematics achievement. As for research question 2, equation F.l and associated hypothesis 
tests were used to generate results for the exhibits for research question 3. 

The achievement analysis comparing magnet schools to their districts for research question 3 
was conducted using achievement z-scores standardized on a base year. A description of this 
approach appears in section F. 1.1. As in the achievement exhibits for research question 1, such 
as Exhibit F.3, the exhibits in this section display results in terms of percentile ranks. A 
description of the conversion of z-scores to percentile ra nks appears in section F.l. 3. 

Exhibit 5.8, reproduced here as Exhibit F.9, presents results for the average EFA percentile 
rank in traditional magnet schools and their districts. The presentation of results is similar to 
the presentation of results for the concentration of students from minority racial/ethnic 
backgrounds in magnet schools and their districts described in section F.2.1.2. First, equation F.l 
was estimated using the annual average EFA z-score in study magnet schools (traditional 
magnet schools in Chapter 5 and destination magnet schools in Chapter 6) as the dependent 
variable. Z-score results were then transformed into percentile ranks for display. The average 
EFA achievement in traditional magnet schools was at the 35.5th percentile before and the 
43.6th percentile after conversion. As in Exhibit 5.3, the pre-post change in the average EFA 
achievement (8.1 percentile points) and significance test are provided in the exhibit between the 
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preconversion and postconversion numbers. The asterisk (*) next to the change indicates that it 
was statistically significant. 

Second, equation F.l was estimated using the annual average ELA z-score in the districts of 
neighborhood magnet schools as the dependent variable. Z-score results were then transformed 
into percentile ranks for the display. The average ELA achievement in the districts of traditional 
magnet schools was at the 51.1th percentile before and 56.8th percentile after conversion. The 
asterisk (*) next to the change (5.6 percentile points) indicates it was statistically significant. 

Third, as described in section F.l. 3, differences in average achievement between magnet schools 
and their districts were calculated as the difference between the displayed percentiles for magnet 
schools and their districts. The preconversion difference in ELA achievement was -15.7 percentile 
points (= 35.5 - 51.1); the postconversion difference in ELA achievement was -13.2 percentile 
points (= 43.6 - 56.8). Like the analyses described in section F.l. 3, all statistical tests were 
conducted using the original z-scores; the arrow symbols and asterisks in Exhibit 5.7 provide 
information about statistical significance: 

■ The asterisk (*) next to the preconversion average difference, -15.7, indicates that before 
conversion the average ELA achievement in magnet schools was statistically different 
from the average in their districts. 

■ The asterisk (*) next to the postconversion average difference, -13.2, indicates that after 
conversion the average ELA achievement in magnet schools was statistically different 
from the average in their districts. 

■ The asterisk (*) next to the change in the difference, 2.5 percentile points, indicates that 
the postconversion average difference was statistically different from the preconversion 
average difference. 

Exhibit F.9. Reprint of Exhibit 5.8. Achievement in Traditional Magnet Schools and 
Their Districts (Average Across Schools) - ELA 


70 



30 


Preconversion Postconversion 


* Magnet 


■ District 
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F.4.2. Investigating the Role of Magnet Conversion 

This section describes the methods for the CITS analyses of achievement in the magnet schools 
for research question 3 in Chapter 5. Like the comparison group of schools described in sections 
F. 2.2.1 and F.3.2, the comparison group of schools used in the achievement analysis is the set of 
elementary public schools in the district that were never charter schools, never magnet schools, 
and in continuous operation throughout the years of the study. The analysis model used for the 
CITS analysis in research question 3, however, differed from that used for similar analyses in 
research questions 2 and 3; details of the model are provided in section F.4.2. 1. The exhibits 
corresponding to the CITS analysis for research question 3, however, were constructed similar to 
those constructed for the CITS analyses in research question 2; a description follows in section 
F.4.2.2. 

For two reasons, the analysis of the association between conversion and achievement of 
neighborhood students was limited to traditional magnet schools and, hence, the CITS analysis 
for research question 3 appears in Chapter 5 but not in Chapter 6. First, the analysis focused on 
neighborhood students because the challenges in predicting how students from outside the 
neighborhood might have performed in the absence of conversion were prohibitive.^^ Second, 
the analysis was limited to traditional magnet schools because neighborhood students in 
destination schools were already higher achieving than average for their districts before 
conversion. At destination schools, the policy interest is largely in the students from outside the 
neighborhood who likely seek attendance at the magnet school because their neighborhood school is 
underperforming . 

F.4.2. 1. The Analysis Model. To examine the association between magnet conversion and 
neighborhood student achievement, the analysis used a CITS approach that predicted the 
achievement for neighborhood students in study magnet schools had the school not converted, 
based on the achievement for similar students in neighborhood schools that did not convert. This 
analysis used the same comparison group as in the analysis for research questions 2 and 3 
(described in sections F.2.2 and F.3.2) but was conducted at the student level rather than the 
school level. Conducting analysis at the student level had two benefits. First, although the 
analysis included all students in the schools, the model was designed to identify the association 
of magnet conversion with changes in achievement for neighborhood students rather than 
students from outside the neighborhood or students for whom a residency status could not be 
determined. Student-level data were necessary to differentiate between neighborhood students 
and students from outside the neighborhood. Second, using student-level data made it possible to 
control for differences in student characteristics across the schools and changes in those 
characteristics over time. The model can more accurately account for changes in these 


The analysis of outcomes for students from outside the neighborhood is more challenging than the analysis of 
neighborhood student outcomes because students from outside the neighborhood have chosen to attend the magnet 
school. Those students from outside the neighborhood may have been higher achieving than other similar students 
because they or their families had prior knowledge that they would benefit most from the specialized theme or 
instruction of the magnet school. When researchers observe only those who will benefit most from an intervention, 
as in this case with students from outside the neighborhood, this may introduce selection bias and make it difficult to 
identify an appropriate comparison group. Analysis of neighborhood students does not suffer from selection bias to 
the same degree because in general neighborhood students did not choose their neighborhood schools. 
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characteristics using the information at the student level rather than using the information 
aggregated to the school level. 

The following model was estimated: 

^Yigt = + I.L1 SsN BHig.t + <PsN 0 N^g.^ + I.U <PsU N K^g^t + 

^NBHP^P^s=m,tTPP^Tigst + ^NON^ONg^-^j-TREATiggf- + 

^UNK^^Xs=m,tPPP^Pigst + ^igst (F-6) 

where 


■ AYigt is the one-year gain in achievement z-score for student i, in grade g, in year t. 

■ Xqigt is a veetor of q student demographie indieators for student i, in grade g, in year t. 

■ A,g is a set of grade indicators. 

■ Fit ... Fit are year indicator variables for Years 2 through 6, respectively. 

■ NBHigst, NONigst, and UNKtgst represent the dosage for school 5 (proportion of the year 
spent in sehool s) for neighborhood students, students from outside the neighborhood, 
and students with unknown resideney status, respeetively (see Appendix B for diseussion 
of dosages). 

■ NBFls=m,t, NONs=m.t, and UNKs=m,t represent the dosage at eaeh study magnet sehool for 
neighborhood students, students from outside the neighborhood, and students with 
unknown resideney status, respectively. 

■ TREAT igi? = 1 if sehool s is an MSAP school in year t when the program was active; 

0 otherwise. 

The achievement measures used for this CITS analysis were standardized differently than the 
achievement measures used in the comparisons of magnet sehools to their districts. In the 
analysis eomparing magnet schools to their districts, achievement z-scores were standardized 
relative to a base-year (see seetion F.1.1). For the eonstruetion of the one-year gain in 
achievement z-score, AYtgst, achievement z-scores were standardized within year. Hence, the 
achievement gain scores do not show overall growth in achievement but instead show growth 
relative to other students in the district. 

The first summation term in equation F.6 ineludes Q student demographic variables (which are 
further described in Exhibit F.IO). The seeond summation includes grade indicator variables, and 


** Similar to the CITS used to analyze the concentration of racial/ethnic minority students and economically 
disadvantaged students for research question 2 (equation F.3), the CITS model shown in equation F.6 also included 
a trend variable and interaction terms between the trend variables and the TREAT variables. For simplicity, the trend 
variable is not shown in equation F.6, but the results for the trend are reported in Exhibit 1. 19 for traditional 
magnets. As in equation F.3, the coding of the trend variable was designed so that its inclusion in the model would 
not affect the estimate for the TREAT variable. 
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the third through fifth contain the enrollment dosages at each of S schools. Note that the model 
includes separate enrollment dosage coefficients for students depending on the school(s) in which 
they were enrolled and whether they were enrolled as neighborhood students, students from outside 
the neighborhood, or students with unknown residency status. The enrollment dosage variables — 
NBH, NON, and UNK — indicate the proportion of the test-to-test year that student i was enrolled at 
school 5 as a neighborhood student, as a student from outside the neighborhood, or as a student 
with unknown residence status, respectively. The variables Fjt,j = 2, 3,. . .6 are year dummies that 
capture districtwide time trends in a flexible way.^^ The analysis allowed for clustering at the 
school-level based on the school in which each student was enrolled for the greatest part of the test- 
to-test school year. Exhibit F.IO shows the list of student characteristics in the vector X. 


Exhibit F.10. Student-Level Characteristics Included as Control Variables in Equation F.6 


Variable 

Details on Coding 

Age 

Two indicators for greater than and less than the normal two-year age range for 
the grade 

Free or reduced- 
price lunch eligibility 

Indicator for eligibility for free or reduced-price lunch (not available for Districts 
G, J, and K) 

Special education 

Indicator for designation as a special education student — typically students with 
an individualized education program 

ELL 

Indicator for designation as an ELL 

Race or ethnicity 

Indicators for Asian, black, Hispanic, or other 

Gender 

Indicator for females 

Parental education 

Indicators for level of parental education: not high school graduate, high school 
graduate, some college, college graduate, and graduate school (data available 
only for District J) 


The coefficient equation F.6 is the estimated coefficient on the interaction between the 

indicators for neighborhood students and for attending a conversion magnet school after it had 
converted. The coefficient is interpreted as the average change in annual achievement gain 
associated with magnet conversion for neighborhood students. As noted in equation F.6, we 
estimated similar coefficients for those who were enrolled in study magnet schools as for 
students from outside the neighborhood (Qnon) with unknown residence status (Qunk) but 
did not report them. In particular, for students from outside the neighborhood, there is a potential 
bias in the estimate Qnon due to self-selection; Students from outside the neighborhood may 
have been more likely to choose to enroll at an MSAP school if they believed they were likely to 
benefit from the school’s specialized theme or instructional practice. 

F. 4. 2. 1.1 Power Analysis for CITS Estimates. Exhibit F.l 1 shows the post-hoc MDE for the 
average relationship between magnet conversion and FLA and mathematics achievement 


There is no year dummy for Year 1 because no gain in test score is available in that year. We also exclude a year 
dummy in Year 2 because otherwise the sum of the year dummies would be perfectly collinear with the grade 
dummies that already take the place of the intercept. 
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reported in Chapter 5 (Exhibits 5.8 and 5.9 for ELA and mathematics estimates). As described 
in section E.2.2.3.1 for research question 2, the estimated MDE is derived from the empirical 
standard errors from the models multiplied by a factor of 2.8. This calculation is based on a two- 
sided t-test of each coefficient at the 5 percent level, with a power of 0.8. 


Exhibit F.1 1 . MDE for Estimates of the Role of Magnet Conversion in ELA and 

Mathematics Achievement for Neighborhood Students in Traditional 
Magnet Schools (Alpha = 0.05, Power = 0.8) 


Subject 

Standard Error 

MDE 

ELA gain (z-score units) 

0.052 

0.14 

Mathematics gain (z-score units) 

0.047 

0.13 


SOURCE: District administrative data. 


F.4.2.2. Displays. The exhibits reporting results for the CITS analysis of achievement for 
research question 3 in Chapter 5 were constructed in the same manner as the exhibits for the 
CITS analysis for research question 2 (as described in sections F.2.2.4 and F.3.2). A few extra 
steps were required, however, because the analyses were conducted using z-score gains, but the 
exhibits report gains in percentiles. 

In the analysis, an annual gain is defined as the change in a student’s achievement from one grade 
to the next (e.g., from grade 3 to grade 4). One complication in presenting results as a percentile 
gain was that the transformation of a gain in z-score to a gain in percentiles depends on the prior 
achievement level (i.e., the z-score in the starting grade). Transforming an average z-score gain into 
percentile units required the identification of a prior achievement level to use as the basis for 
transforming each measure. To ensure that the relative ranking of average z-score gains, as used in 
the exhibits, was preserved after transforming the z-score gains into percentile gains, it was necessary 
to use the same prior achievement z-score for each point in the graph that was transformed.^' Eor 
simplicity, it was assumed that the prior achievement for each average z-score gain in the first 
column of E.IO was at the middle of the distribution, which would be a prior z-score of 0.0 (or in 
percentile terms, a prior achievement at the 50th percentile). Hence, the annual gain was 
interpreted as change from a prior z-score of 0 to a current z-score of “0 plus the estimated gain.” 

This process is captured in equation E.7: 

Percentile gain = percentile(ZCurrentAchievement) — percentile(ZPriorAchievement) 


As described later in section F.5.2, the CITS estimates reported in Exhibits 5.8 and 5.9 were converted from 
z-score units to percentile gain units. The standard errors and MDEs reported in Exhibit F. 10 are in z-score gain 
units. For comparison purposes, the ELA gain CITS estimate (Exhibit 5.8) in z-score units was -0.033 and the 
mathematics gain CITS estimate was 0.002. 

To illustrate why the z-score for average prior achievement matters, consider a student with a z-score gain of 
0. 1 from third to fourth grade. If the student started with a z-score of 0.0 in third grade, leading to a z-score of 
0. 1 in fourth grade, the student would move from the 50th to the 54th percentile, a gain of 4 percentile points. If the 
student started with a z-score of - 1 .0 in third grade, leading to a z-score of -0.9 in fourth grade, the student would 
move from the 16th percentile in third grade to the 18.4th percentile in fourth grade, a gain of 2.5 percentile points. 
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— percentile(ZGain + ZPriorAchievement) 

— percentile(ZPriorAchievement) 

= 0(ZGain + ZPriorAchievement) — O (ZPriorAchievement) 

= 0(ZGain + 0) - 0(0) 

= O(ZGain) - 50 (F.7) 


where 

■ ZCurrentAchievement = achievement in the current grade in z-scores 

■ ZPriorAchievement = achievement in the prior grade in z-scores (assumed to be 0) 

■ ZGain = the gain from the prior to the current grade in z-scores 

■ 0(. ) = the standard normal CDF function 

Exhibit F.12, reprinted from Exhibit 5.9, displays the ELA achievement results in percentile gain 
units. Exhibit F.13 provides details on how the numbers were derived. The first column of F.12 
contains the numbers as originally calculated using z-score gains. The second column provides 
the derivation for each of the numbers in the first column following the procedure described in 
section F.2.2.4. The third column has the corresponding numbers from the first column 
expressed as percentile gains (as reported in Exhibit F.12). The first three rows of Exhibit F.12 
correspond to the 3 points in the display and the final row contains the CITS estimate (displayed 
as the change associated with conversion in the Exhibit F.12). 

Exhibit F.12. Reprint of Exhibit 5.9. The Role of Magnet Conversion in ELA Achievement 
for Neighborhood Students in Traditional Magnet Schools (Average Across 
Schools) 
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Exhibit F.13. Average Annual Percentile Point Gain in ELA for Neighborhood Students, for Traditional Magnet Schools and 
Neighborhood Schools That Did Not Convert, in z-Score and Percentile Terms 


Component 

Annual z-Score Gain 

Annual Percentile Gain Equivalent 

Number 

Derivation 

Number 

Derivation 

Study magnet schools: 
Preconversion 

0.02 

Equation F.1 with annual z-score gain 
as dependent variable 

0.97 

Computed Based on z-Score Gain: 
O(ZScoreGain) - 50 

Study magnet schools: 
Postconversion 

-0.01 

Equation F.1 with annual z-score gain 
as dependent variable 

-0.45 

Computed Based on z-Score Gain: 
O(ZScoreGain) - 50 

Study magnet schools: 
Postconversion (without 
conversion, predicted) 

0.02 

(Study magnet postconversion) - 
(CITS estimate) 

0.85 

Computed Based on z-Score Gain: 
O(ZScoreGain) - 50 

CITS estimate 

-0.03 

M estimated from equation F.6 

-1.30 

Computed as Postconversion 
Study Magnet — With Conversion - 
Postconversion Study Magnet — 
Without Conversion (Predicted), 
Both in Percentile Units 
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The z-scores shown in Exhibit F.13 were eomputed exactly as described in Exhibit E.7. 

Equation E.7 was then used to transform the first three z-score values shown in Exhibit E.13 into 
percentiles. 

The CITS estimate shown in the final row of Exhibit E.13 was converted to percentiles by 
subtraction rather than by using equation E.7. The CITS estimate, by design, is the difference 
between the magnet schools postconversion actual average gain and the predicted average gain 
had the schools not converted. Hence, once these two numbers were transformed from z-scores 
into percentiles, the difference between them provided the CITS estimate in percentile form. In 
Exhibit F.I2, the CITS estimate appears as an annual percentile gain of -1.3 percentile points, 
obtained by subtracting the magnet school predicted postconversion average (0.85) from the 
magnet school actual postconversion average (-0.45). 

F.5. Potential for Bias in the CITS Estimates for Concentrations of 
Students From Minority Racial/Ethnic and Economically 
Disadvantaged Backgrounds 

A somewhat unusual feature of the CITS estimate for student diversity is that movements of 
students from the other neighborhood public schools to the magnet school also affect changes in 
the dependent variable — AVD in equation F.3 — for the other neighborhood schools (schools in 
the district that did not convert to magnet schools). Magnet schools by design are intended to 
draw students from other schools in the district. If magnet schools are successful in drawing such 
students, one would expect a change in the composition of other neighborhood schools. Thus, if 
the magnet schools are effective, the difference across time for the other neighborhood schools 
will not quite equal zero, all else held constant, as assumed by the study’s CITS methods. 

This possible impact on other neighborhood public schools can create a bias in the estimator 
because the motivation for using CITS is that any change in other neighborhood schools would 
be wholly unrelated to the change in study schools. It is easiest to show the source of the bias by 
calculating the CITS as a difference in differences as shown in Exhibit F.I4. The regression 
framework that we actually used (equation F.3) was slightly different than the simplification in 
Exhibit F.14 (equation F.3 controlled for year effects, whereas calculations in Exhibit F.14 do 
not), but it produced something very similar. 


Exhibit F.14. Schematic Diagram of How the CITS Estimate Is Calculated as a Difference 
in Differences 
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To restate the problem in terms of the notation used in Exhibit F. 14, the model calculates A - B 
correctly, but C~ D will be shown to be a second-order bias because, in the postconversion 
period, the proportion of students from minority racial/ethnic or economically disadvantaged 
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backgrounds in the other neighborhood sehools is ehanged slightly by movements of students to 
the study magnet sehool. 

Suppose that in the absenee of the magnet sehool there would be no distrietwide trends in the 
average aeross other neighborhood sehools of the dependent variable — that is, in the absolute 
value of the differenee between the proportion of raeial/ethnie minority students, for example, in 
eaeh neighborhood sehool and in the district. In this case, we would obtain C - D = Instead, 
however, if the magnet school is effective, C - D may be biased away from zero beeause of the 
impaet of the study magnet sehool on the other neighborhood sehools’ set of students. 

The magnitude of the bias in C - Z) ean be ealeulated with the help of some notation. Let there 
be N other neighborhood publie sehools, with sehool A^+ 1 being the magnet sehool. Further, let 

■ mi = number of minority students at sehool i 

■ Wi = number of nonminority students at sehool i 

and suppose that in the postconversion period, some number Yi>0 of nonminority students leave 
other neighborhood sehool j,j e(l, N), and enroll in the magnet sehool. 


Let p = proportion of raeial/ethnie minority students in the distriet: 


P = 




(F.8) 


And, let pj = proportion of raeial/ethnie minority students in sehool i: 

TTLi 

Pi = — ^ 

rrii+Wi 


(F.9) 


For sehool i, our measure of absolute value of the differenee between the sehool and the distriet 
is as follows: 

ABSi = |pi-p| (F.IO) 


Now, suppose the following: 

■ In the study magnet school, school N+ \,yj nonminority students are attracted to the 
sehool after eonversion from sehool j G (1, iV) (i.e., yy nonminority students move from 
sehool j to the study sehool N+ \). 

■ No other students ehange sehools. 

■ The proportion of raeiahethnie minority students in the distriet does not ehange. 


If there were time trends in the average value across schools of our measure of imbalance, it would complicate the 
calculations slightly but would not change the result, under the assumption maintained in a CITS analysis of parallel 
trends in the comparison and treated schools. In such a case, the change in the later period, say, Q, would be added 
to both A and C and therefore would be first-differenced out in the calculation of (A- B)~ (C~ D). Similarly, our 
assumption that a single school sends all the students to the magnet school does not matter. As long as all sending 
schools are either greater or less than the district concentration of raciahethnic minority students, the result will be 
identical to that shown here. 
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The CITS estimate of the ehange in diversity is as follows: 

^CITS~ ^N+1 ~ ^Comparison (F-H) 

where the first term isA~B, showing the ehange at the study magnet sehool, and the seeond 
term subtraets C - D, whieh represents the ehange at the eomparison sehools. 

At the study magnet sehool A + 1 , 

^N+i — — \pN+i,t+i ~ p\~ \PN+i,t ~ p\ 

_ WtJV+l 

mjv+i+wjv+1+7; 


niN+i 

mw+i+wjv+i 


(F.12) 


There are two oases for An+i : 

■ Case 1 (typioal ease): pAf+i,( > p (i.e., the study magnet sehool has a higher proportion of 
students from minority raoial/ethnio baokgrounds than the distriot) 


TTT-N+1 


TTT-N+1 

m^v+i + W;v+i + Yj 


^W+1 + ^N+1 

^W+l 

n - 


rriN+1 \ 

n \ 


m^v+i + W;v+i + Yj 


-mN+iYj 

(mjv + 1 +WJV+ 1 +Y ;■) ("iN+l +'^N+ 1 ) 


< 0 


(F.13) 


Thus imbalanoe falls at the study magnet sehool when jj nonminority students move from 
sehool j to the study sehool A + 1 . (This ignores the unlikely possibility that so many 
nonminority students transfer that the study magnet sehool now has a lower value for 
proportion of raeiakethnie minority students than the distriet.) 

■ Case 2 (atypieal ease): pw+i,/ < p (i.e., the study magnet sehool has equal or lower 
proportion of raeiakethnie minority students than the distriet) 

In this ease, 


^w+i — 


m-N+iYi 

(mjv+i +WJV+ 1 +7;) (wijv+ 1 +WJV+ 1) 


> 0 


(F.14) 


Thus the absolute value of the differenee rises at the study magnet sehool when yj 
nonminority students move from sehool j to the study sehool A + 1 . 


Next, the key question is the sign of -Acompanson in equation F.l 1: 

^ClTS~ ^N+1 ~ ^Comparison 
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It is assumed that the average sehool deviation from the distriet mean has not shown a 
trend, so that C - Z) = 0 is what should be obtained from the other neighborhood sehools 
were there no bias. But the aetual value will differ due to some nonminority students 
leaving school j for the study magnet school. Hence, the sign and magnitude of 
-lS.comparison iu this cxamplc will constitute the bias. 

Let there be an indicator variable — SENDING%MIN 

• = 1 if the sending school has a proportion of racial/ethnic minority students less than 
or equal to the district average 

• = -1 if the sending schools has a proportion of racial/ethnic minority students greater 
than the district average 

Proposition: The value of C - D depends on the numbers of minority and nonminority 
students at sending comparison school j, the number of nonminority students who 
transfer, and, importantly, the number of other neighborhood schools. The magnitude of 
bias in the estimated association between change in imbalance and magnet conversion 
will be as follows: 

BlAS = -AriTs= SENDING%MIN- ^ (F.15) 

N (rtij+w j)(mj+w j~Y j) 

Thus the sign of the bias in the difference in difference estimated is the same as the sign 
ofSENDING%MIN. 


Proof: The bias is given by the extent to which the change in the other 
neighborhood pubiic schoois deviates from zero. 

^Comparison~ ~ P| ~ \Pi,t ~ P|} ~ ^{|Py,t+l ~ P| ~ \Pj,t ~ P|} (F-16) 

because only school j undergoes a change. 

■ Case r (typical case): pj,t < p (i.e., sending neighborhood school has a lower proportion 
of racial/ethnic minority students than the district) 


A 


Comparison 


-{(p - pyt+l) - (P - Pj,t)] 


1 

{ 

mj ] 

n' 

frij + Wj 

rrij + Wj - Yj j 


^ 1 ( -Yjrrij 

N [fmj+w jfmj+w j-y j) 


< 0 


(E.17) 


Thus school j has become less imbalanced as the proportion of racial/ethnic minority 
students increases. 
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But in Acits = An+ i - Acomparison, changcs at school j imply that Acits > An+ i , because 
-Acomparison > 0. There is a positive bias equal to -Acomparison. 

■ Case 2' (atypieal ease): pj.t > p (i.e., eomparison sending sehool has a higher proportion 
of raeial/ethnic minority students than the district) 


^Comparison 


^ 1 [ yj^j ) 

N \{mj+wj)(mj+wj-Yj)) 


> 0 


(F.18) 


School j has become more imbalaneed as the proportion of raeial/ethnie minority students 
inereased even greater than the district average, but -Acomparison < 0 indieates a drop in 
imbalanee. There is a negative bias equal to Acomparison. 


Putting the two oases together leads direotly to equation F.12. 


The most important insight from this proposition is that the bias is proportional to 
1/7V, On average, in the study data, N = 65, 

Here is a numerioal example. Let initial enrollment at the sending sehool j be as follows: 

• fUj = number of minority students at sehool y =150 

• Wj = number of nonminority students at school y = 300 

And let initial enrollment at the study magnet sehool A+ 1 be as follows: 

• mN+\ = number of minority students at sehool A+ 1 = 300 

• WN+i = number of nonminority students at school A+ 1 = 150 

In our actual sample, the increase in the share of students who are from outside the 
neighborhood at the magnet sehool is about 7.5 pereent on average. If the magnet sehool 
has 450 students with 300 minority and 150 nonminority students, then about 
34 nonminority students (about 7.5 pereent of 450) would transfer from sehool y. 

In this ease. 


BIAS = -A 


Comparison' 


-mjYj 


N [{mj + Wj){rrLj + wj - Yj), 


1 

65 


{ 


34*150 

( 150 + 300)(150 + 300 - 34 ) 


} = 0.000419 


At the magnet sehool, N + 1 , then. 


^w+i — 


-mN+iYj 

(mjv+i +WJV+1 +7 ;)(mjv+i +wjv+i) 


- 34*150 

(300 + 150 + 34)(300 + 150 ) 


-0.02342 


(F.19) 


(F.20) 
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If BIAS = 0.000419 and the true effeet (A - B) = 0.02342, then the absolute value of the 
bias as a pereentage of the true effeet is 

100 X ^ 1 . 790/0 (F.21) 

In this ease, the bias is small. 

More generally, one ean ealeulate the absolute value of the bias as a proportion of the 
overall “true” CITS estimate by dividing the absolute value of the bias in equation F.12 
by the absolute value of the expression for the magnet sehool ehange in equation F.13 or 
F.14. Call this proportional bias PropBias: 


PropBias = 


1 

N 


m-Yj 

imj+WjXrrij+Wj-Yj) 

™-N+lYj 

(mjv+i+wjv+i+K;)(mjv+i+wjv+i) 


(F.22) 


This equation makes elear that a variety of parameters will affeet the bias as a proportion 
of the actual change in the magnet school: the number of nonminority students who 
transfer, the number of minority and nonminority students at the sending school,/, and at 
the magnet school, A + 1 . But most important by far, the absolute value of the bias as a 
proportion of the actual magnet school change in imbalance is proportional to UN, where 
N is the number of other neighborhood public schools. The movement of nonminority 
students decreases the minority share by roughly N times as much at the magnet school as 
it increases the minority share on average at the N other neighborhood schools. This is the 
essential reason why the bias will be limited. 

F.6. Potential Explanations for the ELA Achievement Results 
Presented in Chapter 5 

In Chapter 5, the analysis of ELA achievement levels (Exhibit 5.8) found that achievement levels 
in traditional magnet schools increased after conversion relative to their districts, while the 
analysis of annual ELA achievement gains (Exhibit 5.9) found a negative, but not significant, 
association between conversion and neighborhood students’ annual gains in ELA achievement. 
One hypothesis that might explain this pattern of results was provided in the chapter: the 
traditional magnet schools might have attracted academically stronger neighborhood students to 
the schools after conversion, who otherwise would have attended other school choice options. 

It is difficult to test this hypothesis, because it would require data on students’ achievement 
before they entered magnet schools which we cannot have for most students because not all 
grades were assessed (e.g., we typically can only observe three years of test scores for students). 
Three other potential explanations were explored, however, pertaining to differences in the way 
the samples were defined for the analysis of achievement levels and gains. 

1. The analysis of achievement levels in Exhibit 5.8 compared magnet schools to all schools 
in the district, including other magnet and charter schools, while the analysis of gains in 
Exhibit 5.9 included only neighborhood schools that did not convert. 
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2. The comparison of achievement levels in magnet schools to the district in Exhibit 5.8 
included all students, while the comparison of gains to students in neighborhood schools 
in Exhibit 5.9 was specific to neighborhood students. 

3. The sample of students with an annual gain score differed from the sample of students 
with a current year test score, because gain scores require the students to have both a 
current year score and a prior year score, and some students were missing a prior year 
score. 

To test whether these differences in the sample might have accounted for the pattern of results 
observed, the relevant analyses were conducted using a common sample. Eor example, to test 
explanation 1 , the average preconversion and postconversion annual achievement gains for 
magnet schools were compared to their districts, the same comparison group used in the analysis 
in Exhibit 5.8. The result was that even when ELA achievement gains in magnet schools were 
compared to ELA achievement gains in their districts, a similar difference in results was found: 
While the analysis of ELA levels showed an increase in magnet schools relative to their districts, 
the analysis of ELA gains showed a decrease relative to their districts. Eor each of the 
explanations listed, the difference in results was similar when a common sample was used, 
implying that differences in the sample do not explain the observed patterns. 
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Appendix G: Subgroup Analyses 

G.1. Variation in the Relationship Between Magnet Conversion and 
Student Achievement by Demographic Subgroup 

The components of the magnet conversion implemented at a school, such as the specialized 
curricula or instructional practices, may have different relationships with achievement for different 
types of students. To supplement the achievement CITS analysis reported in Chapter 5, the study 
examined whether the estimated effects of conversion might differ for the following subgroups; 

■ Minority and nonminority students 

■ Disadvantaged and nondisadvantaged students 

■ Students with disabilities (SWDs) and students who were not SWDs 

■ Students who were ELLs and students who were not ELLs 

This analysis was conducted for traditional magnet schools only, because the number of 
destination schools (four) was too small to support the analysis. 

To examine results for subgroups, equation G.l was used, a modification of the main 
achievement CITS regression model (equation E.6 in Appendix E. 4.2.1). The model was 
estimated separately for each of the four sets of subgroups listed previously. In equation G.l, 51 
indicates one of the subgroups in each pair listed earlier (e.g., the minority subgroup) and S2 
indicates the other (e.g., the nonminority subgroup). 

Q 5 s s s 

= I Pq^qigt + SgRESlggf + ^ ^ ' (PsUNKlggf + 

q=l g=2 s=l s=l s=l 

+ ^ 2^41 + ^ 3^51 + + 

^REsPPPs=m,tPPP^Pigst + ^NONp^^^s=m,t'PPP^'Pigst + ^UNK^P^Ps=m,t'PPP^'Pigst 
^RES PPPs=m,t'PPP^'PigstP2.igst T ^non ^^^s=m,tPPP^PigstP2.igst + 

UNKs=m.JPEATigstS2igst + Sigst (G. 1) 

where 

■ S2jgst = 1 if student 5 is in subgroup 52 and 0 otherwise^^ 

■ ~ the estimate of the relationship of change in achievement gain and magnet 
conversion for neighborhood students in subgroup 51 

■ = the estimated difference between the relationship for subgroup 51 and 
subgroup 52 


Indicators for the subgroups also are included among the demographic variables in the Xqigst term as discussed in 
Appendix F. 
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Separate estimates of the assoeiation between aehievement and magnet eonversion were obtained 
for subgroup S2 by modifying equation G.l to use as the interaetion variable: 

AVigt = PgX^igt + Z^=2 ^g + 2f=i S.REStg,, + Sf=i (psNONtg,, + Sf=i (PsUNKtgst + 

^RES^^^s=m,t'^^^^'^igst + ^NiN^^^s=m,t'^^^^'^igst + ^UNK^XIKg^y^i-TREATiggi- + 
^^^s=m,tT'REATigsfSligsf + B^g^^NONs^^fTREATigsfSligsf + 

6f;il^^UNK,=rn.tTREATtgstSligst + Sigst (G.2) 


where 

■ Sljgst = 1 if student 5 is in subgroup and 0 otherwise 

■ 0II5 = the estimate of the relationship of ehange in aehievement gain and magnet 
eonversion for neighborhood students in subgroup S2 

In equations G.l and G.2, the eoeffieients ~ construetion. 

Analysis of results aeross all traditional study magnet sehools was eondueted using the same 
meta-analysis teehniques used for results reported in Chapters 5 and 6 and deseribed in Appendix 

F. 2.2.3. The meta-analysis was eondueted separately first using the eoeffieients from eaeh 

distriet and then the eoeffieients. 

G. 2. Results by Demographic Subgroup 

Meta-analysis results by subgroup are presented in Exhibit G.l. The estimated relationship 
between magnet eonversion and aehievement gains for neighborhood students was not 
statistieally signifieant for any subgroup. The differenee between subgroups in the relationship 
between magnet eonversion and aehievement gains for neighborhood speeial edueation students 
eompared to nonspeeial edueation students was statistieally signifieant. However, beeause the 
estimates were not signifieant for either subgroup, this finding was not explored further or 
ineluded in the body of the report. 
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Exhibit G.1. Estimates of the Role of Magnet Conversion in ELA and Mathematics 

Achievement for Neighborhood Students in Traditional Magnet Schools, by 
Subgroup and Difference Between Complementary Subgroups 


Subgroup 

ELA 

Estimate (p-value) 

Mathematics 
Estimate (p-value) 

Minority 

-0.03 (0.504) 

# (0.999) 

Nonminority 

0.06 (0.457) 

0.05 (0.122) 

Difference 

-0.09 (0.353) 

-0.06 (0.067) 

Disadvantaged 

-0.06 (0.357) 

-0.01 (0.851) 

Nondisadvantaged 

-0.05 (0.550) 

-0.05 (0.466) 

Difference 

# (0.974) 

0.02 (0.515) 

ELL 

-0.08 (0.200) 

-0.04 (0.407) 

Non-ELL 

-0.03 (0.667) 

0.01 (0.846) 

Difference 

-0.06 (0.149) 

-0.05 (0.108) 

Special education 

-0.07 (0.295) 

0.06 (0.406) 

Nonspecial education 

-0.03 (0.564) 

# (0.925) 

Difference 

-0.06* (0.001) 

0.05 (0.541) 


* = statistically significant (p < 0.05). # = rounds to zero. 

NOTE: ELA and mathematics achievement measures are annual z-score gains. N = ^7 study magnet schools in 10 districts except 
for analysis of disadvantaged and nondisadvantaged where N = ^^ study magnet schools in seven districts. 

SOURCE: District administrative data. 
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Appendix H: Sensitivity Anaiyses for CiTS Findings 

H.1. Sensitivity Analysis for Diversity CITS Findings 

The theory of aetion underlying magnet eonversion eontends that schools undergoing magnet 
conversion will be able to attract students from outside the neighborhood attendance zone, which 
will increase the schools’ racial/ethnic or socioeconomic diversity. However, the theory of action 
depends on the assumption that a pool of students exists elsewhere in the district who could be 
attracted to the magnet schools and make the schools more diverse. 

As shown Exhibit 3.1, three traditional magnet schools were in study districts (one in district A, 
one in district B for the 2004 cohort, and one in district B for the 2007 cohort) that had a high 
proportion of students from minority racial/ethnic backgrounds in the year prior to conversion 
(more than 80 percent), which may have made it difficult for the study magnet schools in these 
districts to attract large numbers of nonminority students because there were few such students to 
attract. In addition, one destination study school was in a district (district F) that had a low 
proportion (less than 20 percent) of racial/ethnic minority and disadvantaged students. The 
sensitivity of analysis results to the inclusion of these schools in the study was investigated by 
conducting the analyses for research questions 2 and 3 in Chapters 5 and 6 after dropping these 
schools from the study sample. 

Exhibit H.l shows that dropping these schools from the analysis did not change the pattern of 
statistically significant results, except in one case, discussed below. For the concentration of 
minority students in traditional magnet schools, the results of the primary and sensitivity 
analyses were similar: magnet conversion was statistically significantly associated with bringing 
the proportion of students from a racial/ethnic minority background in the traditional magnet 
schools closer to the district proportion. For the proportion of students from economically 
disadvantaged backgrounds in traditional magnet school, a sensitivity test was not conducted 
because no conversion magnet schools were located in districts that had a high proportion of 
economically disadvantaged students. 

For the proportion of racial/ethnic minority students in destination magnet schools, the estimated 
association between magnet conversion and change in the concentration of minority students was 
not statistically significant in either the primary or sensitivity analysis. For the proportion of 
students from economically disadvantaged backgrounds in destination magnet schools, the 
primary estimate (including all four destination schools) was negative and not statistically 
significant. The sensitivity estimate (which removed one of the destination schools) also was 
negative, but larger in magnitude and statistically significant, indicating that magnet conversion 
was associated with bringing the proportion of economically disadvantaged students in the 
destination magnet schools closer to the district proportion. 
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Exhibit H.1. Primary Estimates of the Role of Magnet Conversion in the Concentration of Racial/Ethnic Minority and 

Economically Disadvantaged Students (as Reported in Chapters 5 and 6) and Results of Sensitivity Analyses 
Dropping Schools in Districts With Small Populations of Students Whom the Schools Were Trying to Attract, 
for Traditional and Destination Schools 


Analysis 

Traditional (Chapter 5) 

Destination (Chapter 6) 

Absolute Value of the 
Difference in Minority 
Concentration 
Estimate (p-value) 

Absolute Value of the 
Difference in 
Disadvantaged 
Concentration 
Estimate (p-value) 

Absolute Value of the 
Difference in Minority 
Concentration 
Estimate (p-value) 

Absolute Value of the 
Difference in 
Disadvantaged 
Concentration 
Estimate (p-value) 

Primary estimate (%) 

-1.90* (0.01) 

-2.51 (0.12) 

-1.14 (0.54) 

-5.68 (0.07) 

Sensitivity analysis — Dropping schools in 
districts with small populations of students 
whom the schools were trying to attract (%) 

-2.25* (0.04) 

(t) 

1.22 (0.20) 

-7.82* (0.01) 


* = statistically significant with p < 0.05, two-tailed test, t = Not applicable; no schools were excluded in sensitivity analysis. 


NOTE: For traditional schools: W = 17 study magnet schools in 10 districts for the AVDm primary estimate and A/ = 14 study magnet schools in eight districts for the AVDm sensitivity 
analysis; N = 1 1 study magnet schools in seven districts for the AVDd primary estimate. For destination schools: N = 4 study magnet schools in three districts for the AVDm primary 
estimate and N = 3 study magnet schools in two districts for the AVDy sensitivity analysis; N = 4 study magnet schools in three districts for the AVDq primary estimate and A/ = 3 study 
magnet schools in two districts for the AVDq sensitivity analysis. AVD is the absolute value of the difference between a school’s concentration and the district's concentration. 

SOURCE: District administrative data. 
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H.2. Sensitivity Anaiysis for Achievement CiTS Findings 


The achievement CITS analysis (research question 3 in Chapter 5, comparison to neighborhood 
schools in the district that did not convert) incorporated student dosage variables that measured 
the proportion of the year that students were enrolled in each school within the district. To check 
the sensitivity of study results to this specification, a sensitivity analysis was conducted in which 
students were assigned to only one school per year based on their school of maximum enrollment 
dosage. Modifying the model to allow for this change involved replacing the school-specific 
enrollment dosage variables RES, NON, and UNK in the original equation (see equation F.6 in 
Appendix F.4.2.1) with 0/1 dummy indicators RESDUM, NONDUM, and UNKDUM denoting 
which school a student attended longest each year as a neighborhood student, student from 
outside the neighborhood, or student with unknown residency, respectively. Equation H. 1 
provides the model used for this sensitivity check: 


^^igt — Tiq=^PqRqigt + T,g=2^g + 


I 

S = 1 


I 


s 

V 


S,RESDUM^^+ > cpNONDUM^g.t + + 


'igst ■ ^ T-s- 
s=l 


S =1 


^1^3t + + ^S^7t + 

^UNK^^^^^Ms=m,tTREATigst + Sigst 


(H.l) 


where 


■ Qres = estimate of the average relationship of change in achievement gain and magnet 
conversion for neighborhood students. 

The results of the primary analysis and the sensitivity analysis for student achievement appear in 
Exhibit H.2. The alternative specification did not change the results. Eike the primary estimates, 
in the sensitivity analysis, the estimated relationship between conversion and change in the annual 
achievement gain in EEA and mathematics was close to zero and not statistically significant. 
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Exhibit H.2. Primary Estimates of Role of Magnet Conversion in ELA and Mathematics 
Achievement for Neighborhood Students in Traditional Magnet Schools (as 
Reported in Chapters 5) and Results of Sensitivity Analyses for 
Achievement Outcomes Where Students Were Assigned to Only One 
School per Year 


Analysis 

ELA 

Estimate (p-value) 

Mathematics 
Estimate (p-value) 

Primary estimate (z-score equivalent of results 
reported in Chapter 5) 

-0.03 (0.52) 

#(0.97) 

Sensitivity analysis — assigning students to school of 
maximum enrollment dosage (percentile points) 

-0.03 (0.67) 

0.01 (0.87) 


* = statistically significant with p < 0.05, two-tailed test. 

NOTE: # = Rounds to zero. N = M study magnet schools in 10 districts. The dependent variable was the annual change in a 
student’s achievement z-score. 

SOURCE: District administrative data. 
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Appendix I: Supporting Exhibits for Chapter 5 

This appendix provides additional details for the results presented in Chapter 5. For each exhibit 
in Chapter 5, the appendix contains one or more supporting exhibits with the estimates in 
Chapter 5 and />-values for each statistical test. In addition, for the analyses comparing change in 
magnet schools and neighborhood public schools (CITS analysis), two additional supporting 
exhibits are provided. The first (e.g.. Exhibit 1.5) contains baseline averages showing the mean of 
the key variables (percentage minority, percentage disadvantaged, average ELA and mathematics 
annual gain) for each magnet conversion school and the other neighborhood schools in its district 
for the year just before conversion. The second (e.g.. Exhibit 1.7) is a “forest” plot of results from 
the meta-analysis. The plot displays the estimate of the association between magnet conversion 
and change in the outcome for each district separately and for the average across districts, along 
with 95-percent confidence intervals. The final section provides the estimates of the trend over 
the postconversion period in the association between outcomes and magnet school conversion, 
which were estimated as part of each CITS model (described in a footnote to equation F.3 in 
section F.2.2.2 for the diversity CITS analysis and a footnote to equation F.6 in section F.4.2.1 
for the achievement CITS analysis). 

Exhibits I.l, 1.2, and 1.3 support exhibits in Chapter 5 for research question 1. Exhibits 1.4 
through 1.7 support exhibits in Chapter 5 for the research question 2 analysis of the concentration 
of racial/ethnic minority students. Exhibits 1.8 through I.l 1 support exhibits in Chapter 5 for the 
research question 2 analysis of the concentration of economically disadvantaged students. 
Exhibits 1.12 through 1.18 support exhibits in Chapter 5 for research question 3. Exhibit 1.19 
reports CITS trend estimates. 

Supporting Exhibits for Research Question 1 


Exhibit 1.1. Share of Students From Outside the Neighborhood in Traditional Magnet 
Schools, Pre- and Postconversion (Average Across Schools) 


Share of Students From Outside the Neighborhood (%) 

Preconversion 

Estimate 

Postconversion 

Estimate 

Change 

Estimate (p-vaiue) 

21.0 

26.8 

5.8* (0.000) 


* = statistically significant (p < 0.05). 
NOTE: N = M schools in 10 districts. 
SOURCE: District administrative data 
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Exhibit 1.2. Characteristics of Neighborhood Students and Students From Outside the 
Neighborhood in the Traditional Magnet Schools (Average Across Schools) 


a. Proportion of Students From Minority Racial/Ethnic Backgrounds (%) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Neighborhood students 

83.0 (t) 

84.0 (t) 

1.0 (0.333) 

Students from outside the 
neighborhood 

84.4 (t) 

84.4 (t) 

# (0.987) 

Difference 

1.4 (0.405) 

0.5 (0.698) 

-1.0 (0.645) 


b. Proportion of Students From Disadvantaged Backgrounds (%) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Neighborhood students 

70.1 (t) 

73.8 (t) 

3.7* (0.009) 

Students from outside the 
neighborhood 

65.4 (t) 

71.2 (t) 

5.8* (0.008) 

Difference 

-4.7* (0.026) 

-2.6 (0.072) 

2.1 (0.403) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: # = Rounds to zero. Differences were calculated using unrounded numbers and may not equal differences calculated from 
numbers shown in the table. For the proportion of students from minority racial/ethnic backgrounds, N = M schools in 10 districts; 
for the proportion of students from economically disadvantaged backgrounds, N = 1 1 schools in seven districts. 

SOURCE: District administrative data. 
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Exhibit 1.3. Achievement of Neighborhood Students and Students From Outside the 
Neighborhood in Traditional Magnet Schools (Average Across Schools) 


a. ELA Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Neighborhood students 

37.1 (t) 

44.4 (t) 

7.3* (0.000) 

Students from outside the 
neighborhood 

35.6 (t) 

44.0 (t) 

8.4* (0.012) 

Difference 

-1.5 (0.545) 

-0.4 (0.854) 

1.1 (0.697) 


b. Mathematics Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Neighborhood students 

39.2 (t) 

48.1 (t) 

8.9* (0.000) 

Students from outside the 
neighborhood 

35.1 (t) 

48.1 (t) 

13.0* (0.000) 

Difference 

-4.1* (0.048) 

# (0.903) 

4.1 (0.091) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: # = Rounds to zero. Differences were calculated using unrounded numbers and may not equal differences calculated from 
numbers shown in the table. N= 17 schools in 10 districts. 

SOURCE: District administrative data. 

Supporting Exhibits for Research Question 2 - Concentration of 
Raciai/Ethnic Minority Students 


Exhibit 1.4. Concentration of Racial/Ethnic Minority Students in Traditional Magnet 
Schools (Average Across Schools) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet School (%) 

84.5 (t) 

84.9 (t) 

0.4 (0.565) 

District (%) 

64.1 (t) 

66.3 (t) 

2.3* (0.000) 

Difference (%) 

20.4* (0.000) 

18.5* (0.000) 

-1.9* (0.008) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. A/= 17 schools in 10 districts. 

SOURCE: District administrative data. 
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Exhibit 1.5. Percentage of Students From Minority Racial/Ethnic Backgrounds 

for the Magnet School and Other Neighborhood Schools in the Year Before 
Conversion, by Traditional Magnet School 




Magnet 

Other Neighborhood Schoois (Combined) 

wGIlOOl 

Percentage 

Standard Deviation 

Percentage 

Standard Deviation 

A001 

96.4 

18.6 

89.1 

31.2 

B002 

91.2 

28.4 

88.9 

31.4 

BOOS 

87.8 

32.8 

91.4 

28.1 

C001 

83.6 

37.3 

61.2 

48.7 

D001 

95.8 

20.0 

55.6 

49.7 

D002 

90.8 

29.1 

55.6 

49.7 

D003 

99.3 

8.2 

55.6 

49.7 

F002 

49.5 

50.2 

20.2 

40.2 

F003 

45.3 

50.1 

20.2 

40.2 

G001 

71.9 

45.1 

51.2 

50.0 

G002 

97.4 

15.8 

51.2 

50.0 

H001 

95.5 

20.8 

72.5 

44.7 

1001 

98.8 

10.8 

55.6 

49.7 

J001 

62.0 

48.8 

72.7 

44.6 

J002 

82.1 

38.4 

72.7 

44.6 

K001 

95.0 

21.8 

71.4 

45.2 

K002 

90.7 

29.2 

71.4 

45.2 


NOTE: A/= 17 schools in 10 districts. 
SOURCE: District administrative data. 


Exhibit 1.6. The Role of Magnet Conversion in the Concentration of Racial/Ethnic 

Minority Students in Traditional Magnet Schools (Average Across Schools) 


Difference Between Schoois and Their Districts (%) 

Component 

Estimate 

p-vaiue 

Change associated with conversion from 
CITS 

-1.9* 

0.010 

Percentage of variation in results across 
districts that is not due to error (P) 

89.1 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: A/ = 17 schools in 10 districts. CITS = comparative interrupted time series.. 
SOURCE: District administrative data. 
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Exhibit 1.7. Plot for Meta-analysis of the Role of Magnet Conversion in the 

Concentration of Racial/Ethnic Minority Students in Traditional Magnet 
Schools 
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NOTE: A/ = 1 0 districts. 

SOURCE: District administrative data. 
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Supporting Exhibits for Research Question 2 - Concentration of 
Economicaiiy Disadvantaged Students 


Exhibit 1.8. Concentration of Economically Disadvantaged Students in Traditional 
Magnet Schools (Average Across Schools) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet School (%) 

71.4 (t) 

74.2 (t) 

2.8* (0.015) 

District (%) 

46.1 (t) 

50.1 (t) 

3.9* (0.000) 

Difference (%) 

25.3* (0.000) 

24.1* (0.000) 

-1.2 (0.278) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. A/ = 1 1 schools in seven districts. 

SOURCE: District administrative data. 


Exhibit I.9. Percentage of Students From Economically Disadvantaged Backgrounds 
for the Magnet School and Other Neighborhood Schools in the Year Before 
Conversion, by Traditional Magnet School 



Magnet 

Other Neighborhood Schools (Combined) 

wOl lOOl 

Percentage 

Standard Deviation 

Percentage 

Standard Deviation 

A001 

86.7 

33.6 

68.9 

46.1 

B002 

63.9 

48.1 

45.7 

49.8 

B003 

59.5 

48.9 

43.7 

49.5 

C001 

90.9 

29.0 

75.2 

43.2 

D001 

94.4 

22.7 

46.9 

49.8 

D002 

77.2 

41.5 

46.9 

49.8 

D003 

93.2 

25.3 

46.9 

49.8 

F002 

45.0 

48.6 

11.6 

31.3 

F003 

25.6 

42.8 

11.6 

31.3 

G001 

— 

— 

— 

— 

G002 

— 

— 

— 

— 

H001 

79.1 

40.6 

48.7 

50.0 

1001 

84.9 

35.4 

48.3 

49.5 

J001 

— 

— 

— 

— 

J002 

— 

— 

— 

— 

K001 

— 

— 

— 

— 

K002 

— 

— 

— 

— 


— = Not available; data were not collected or not reported. 
NOTE: A/ = 1 1 schools in seven districts. 

SOURCE: District administrative data. 
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Exhibit 1.10. The Role of Magnet Conversion in the Concentration of Economically 

Disadvantaged Students in Traditional Magnet Schools (Average Across 
Schools) 


Difference Between Schools and Their Districts (%) 

Component 

Estimate 

p-vaiue 

Change associated with conversion from 
CITS 

-2.5 

0.118 

Percentage of variation in results across 
districts that is not due to error (P) 

95.5 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: N = 1 1 schools in seven districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 


Exhibit I.1 1 . Plot for Meta-analysis of the Role of Magnet Conversion in the Concentration 
of Economically Disadvantaged Students in Traditional Magnet Schools 
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NOTE: N = seven districts. 

SOURCE: District administrative data. 
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Supporting Exhibits for Research Question 3 

Exhibit 1.12. Achievement in Traditional Magnet Schools and Their Districts (Average 
Across Schools) 


a. ELA Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet School 

35.5 (t) 

43.6 (t) 

8.1* (0.000) 

District 

51.1 (t) 

56.8(t) 

5.6* (0.000) 

Difference 

-15.7* (0.000) 

-13.2* (0.000) 

2.5* (0.024) 


b. Mathematics Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet School 

37.6 (t) 

47.6 (t) 

10.0* (0.000) 

District 

52.0 (t) 

60.0 (t) 

8.0* (0.000) 

Difference 

-14.4* (0.000) 

-12.4* (0.000) 

2.0 (0.133) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = M schools in 10 districts. 

SOURCE: District administrative data. 
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Exhibit 1.13. Annual English Language Arts z-Score Gain for the Magnet School and 

Other Neighborhood Schools in the Year Before Conversion, by Traditional 
Magnet School 


School 

Magnet 

Other Neighborhood Schools 
(Combined) 


Average 

Standard Deviation 

Average 

Standard Deviation 

A001 

-0.142 

0.604 

-0.082 

0.616 

B002 

-0.045 

0.458 

-0.027 

0.523 

BOOS 

-0.085 

0.447 

0.031 

0.513 

C001 

0.108 

0.648 

-0.038 

0.571 

D001 

0.012 

0.885 

-0.014 

0.632 

D002 

0.019 

0.654 

-0.014 

0.632 

D003 

-0.211 

0.848 

-0.014 

0.632 

F002 

0.276 

0.545 

0.073 

0.739 

F003 

-0.117 

0.719 

0.073 

0.739 

G001 

0.156 

0.714 

0.083 

0.708 

G002 

-0.002 

0.631 

0.083 

0.708 

H001 

0.015 

0.578 

0.004 

0.548 

1001 

0.322 

0.843 

-0.039 

0.658 

J001 

-0.011 

0.546 

0.007 

0.570 

J002 

0.019 

0.467 

0.007 

0.570 

K001 

0.096 

0.654 

0.127 

0.749 

K002 

0.124 

0.768 

0.127 

0.749 


NOTE: A/= 17 schools in 10 districts. 
SOURCE: District administrative data. 


Exhibit 1.14. The Role of Magnet Conversion in ELA Achievement for Neighborhood 
Students in Traditional Magnet Schools (Average Across Schools) 


Annual Percentile Point Gain 

Component 

Estimate 

p-value 

Change associated with conversion from 
CITS 

-1.30 

0.527 

Percentage of variation in results across 
districts that is not due to error (P) 

99.0 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: A/ = 17 schools in 10 districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 
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Exhibit 1.15. Plot for Meta-analysis of the Role of Magnet Conversion in ELA 

Achievement for Neighborhood Students in Traditional Magnet Schools 
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NOTE: A/ = 10 districts. Estimates in this exhibit are in terms of z-score gains rather than percentile point gains. 
SOURCE: District administrative data. 
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Exhibit. 1.16. Annual Mathematics z-Score Gain for the Magnet School and 

Other Neighborhood Schools in the Year Before Conversion, by Traditional 
Magnet School 




l/lagnet 

Other Neighborhood Schoois (Combined) 

wOl lOOl 

Average 

Standard Deviation 

Average 

Standard Deviation 

A001 

-0.082 

0.791 

-0.074 

0.636 

B002 

-0.101 

0.479 

-0.021 

0.603 

BOOS 

-0.213 

0.641 

0.015 

0.585 

C001 

-0.214 

0.915 

-0.005 

0.600 

D001 

0.081 

0.717 

-0.019 

0.604 

D002 

0.024 

0.672 

-0.019 

0.604 

D003 

-0.171 

0.498 

-0.019 

0.604 

F002 

-0.193 

0.601 

0.091 

0.691 

F003 

-0.244 

0.741 

0.091 

0.691 

G001 

0.145 

0.571 

0.003 

0.657 

G002 

0.043 

0.568 

0.003 

0.657 

H001 

0.165 

0.608 

0.005 

0.661 

1001 

0.488 

0.887 

-0.046 

0.642 

J001 

-0.066 

0.699 

0.002 

0.656 

J002 

0.095 

0.665 

0.002 

0.656 

K001 

-0.064 

0.636 

-0.005 

0.650 

K002 

-0.046 

0.643 

-0.005 

0.650 


NOTE: A/= 17 schools in 10 districts. 
SOURCE: District administrative data. 


Exhibit 1.17. The Role of Magnet Conversion in Mathematics Achievement for 

Neighborhood Students in Traditional Magnet Schools (Average Across 
Schools) 


Annuai Percentiie Point Gain 

Component 

Estimate 

p-vaiue 

Change associated with conversion from 
CITS 

0.08 

0.967 

Percentage of variation in results across 
districts that is not due to error (P) 

97.4 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: A/ = 17 schools in 10 districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 
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Exhibit 1.18. Plot for Meta-analysis of the Role of Magnet Conversion in Mathematics 
Achievement for Neighborhood Students in Traditional Magnet Schools 
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Postconversion Trend Analysis for Chapter 5 

Exhibit 1.19. Trend Over the Postconversion Period for the Role of Magnet Conversion 
in Changes in Traditional Magnet Schools 


CITS Model 

Trend 

Estimate (p-value) 

Minority (percentage point) 

-0.91* (0.004) 

Disadvantaged (percentage point) 

-1.12 (0.197) 

ELA (annual z-score gain) 

-0.006 (0.789) 

Mathematics (annual z-score gain) 

0.015(0.336) 


* = statistically significant (p < 0.05). 

NOTE: N = M schools in 10 districts for minority, ELA, and mathematics analysis. A/ = 1 1 schools in seven districts for 
disadvantaged analysis. CITS = comparative interrupted time series. 

SOURCE: District administrative data. 
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What Happens When Schools Become Magnet Schools? — Appendices 


Appendix J: Supporting Exhibits for Chapter 6 

This appendix provides additional details for the results presented in Chapter 6. For each exhibit 
in Chapter 6, the appendix contains one or more supporting exhibits with the estimates in 
Chapter 6 and />-values for each statistical test. In addition, for the analyses comparing change in 
magnet schools and neighborhood public schools (CITS analysis), two additional supporting 
exhibits are provided. The first (e.g.. Exhibit J.5) contains baseline averages showing the mean 
of the key variables (percentage minority, percentage disadvantaged, average ELA and 
mathematics annual gain) for each magnet conversion school and the other neighborhood schools 
in its district for the year just before conversion. The second (e.g.. Exhibit J.7) is a "forest" plot 
of results from the meta-analysis. The plot displays the estimate of the association between 
magnet conversion and change in the outcome for each district separately and for the average 
across districts, along with 95-percent confidence intervals. The final section provides the 
estimates of the trend over the postconversion period in the association between outcomes and 
magnet school conversion, which were estimated as part of each CITS model (described in a 
footnote to equation E.3 in section E.2.2.2 for the diversity CITS analysis and a footnote to 
equation F.6 in section F. 4.2.1 for the achievement CITS analysis). 

Exhibits J.l, J.2, and J.3 support exhibits in Chapter 6 for research question 1. Exhibits J.4 
through J.7 support exhibits in Chapter 6 for the research question 2 analysis of the concentration 
of racial/ethnic minority students. Exhibits J.8 through J.l 1 support exhibits in Chapter 6 for the 
research question 2 analysis of the concentration of economically disadvantaged students. 
Exhibits 1.12 supports the exhibit in Chapter 6 for research question 3. Exhibit J.l 3 reports CITS 
trend estimates. 

Supporting Exhibits for Research Question 1 


Exhibit J.1. Share of Students From Outside the Neighborhood in Destination Magnet 
Schools, Pre- and Postconversion (Average Across Schools) 


Share of Students From Outside the Neighborhood (%) 

Preconversion 

Estimate 

Postconversion 

Estimate 

Change 

Estimate (p-vaiue) 

34.2 

41.4 

7.3* (0.005) 


* = statistically significant (p < 0.05). 
NOTE: N = four schools in three districts. 
SOURCE: District administrative data. 
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Exhibit J.2. Characteristics of Neighborhood Students and Students From Outside the 
Neighborhood in the Destination Magnet Schools (Average Across Schools) 


a. Proportion of Students From Minority Racial/Ethnic Backgrounds (%) 


Population 

Preconversion 

Estimate 

(p-value) 

Postconversion 

Estimate 

(p-value) 

Change 

Estimate 

(p-value) 

Neighborhood students 

70.9 (t) 

74.6 (t) 

3.7* (0.002) 

Students from outside the neighborhood 

73.6 (t) 

77.1 (t) 

3.5 (0.053) 

Difference 

2.7 (0.088) 

2.5* (0.017) 

-0.2 (0.918) 


b. Proportion of Students From Disadvanaged Backgrounds (%) 


Population 

Preconversion 

Estimate 

(p-value) 

Postconversion 

Estimate 

(p-value) 

Change 

Estimate 

(p-value) 

Neighborhood students 

25.8 (t) 

29.7 (t) 

3.9 (0.087) 

Students from outside the neighborhood 

24.7 (t) 

35.6 (t) 

10.8* (0.011) 

Difference 

-1.0 (0.762) 

5.9* (0.012) 

6.9 (0.095) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = four schools in three districts. 

SOURCE: District administrative data. 


Exhibit J.3. Achievement of Neighborhood Students and Students From Outside the 

Neighborhood in the Destination Magnet Schools (Average Across Schools) 


a. ELA Percentile Rank 


Population 

Preconversion 

Estimate 

(p-value) 

Postconversion 

Estimate 

(p-value) 

Change 

Estimate 

(p-value) 

Neighborhood students 

59.1 (t) 

60.2 (t) 

1.2 (0.550) 

Students from outside the neighborhood 

55.7 (t) 

58.2 (t) 

2.5 (0.520) 

Difference 

-3.3 (0.230) 

-2.0 (0.247) 

1.3 (0.686) 


b. Mathematics Percentile Rank 


Population 

Preconversion 

Estimate 

(p-value) 

Postconversion 

Estimate 

(p-value) 

Change 

Estimate 

(p-value) 

Neighborhood students 

60.2 (t) 

63.3 (t) 

3.1 (0.317) 

Students from outside the neighborhood 

51.4 (t) 

58.3 (t) 

6.9 (0.092) 

Difference 

-8.8* (0.018) 

-5.0* (0.028) 

3.8 (0.359) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = four schools in three districts. 

SOURCE: District administrative data. 
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Supporting Exhibits for Research Question 2 - Concentration of 
Students From Minority Raciai/Ethnic Backgrounds 


Exhibit J.4. Concentration of Racial/Ethnic Minority Students in the Destination Magnet 
Schools (Average Across Schools) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet school (%) 

71.6 (t) 

74.9 (t) 

3.3* (0.001) 

District (%) 

72.0 (t) 

73.8 (t) 

1.8* (0.002) 

Difference (%) 

0.4 (0.512) 

1.1* (0.008) 

0.7 (0.302) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = four schools in three districts. 

SOURCE: District administrative data. 


Exhibit J.5. Percentage of Students From Minority Racial/Ethnic Backgrounds 

for the Magnet School and Other Neighborhood Schools in the Year Before 
Conversion, by Destination Magnet School 


School 

Magnet Schools 

Other Neighborhood Schools 
(Combined) 


Percentage 

Standard Deviation 

Percentage 

Standard Deviation 

B001 

86.9 

33.8 

88.9 

31.4 

B004 

91.5 

27.9 

91.4 

28.1 

E001 

91.7 

27.8 

85.4 

35.3 

F001 

14.7 

35.6 

20.2 

40.2 


NOTE: N = four schools in three districts. 
SOURCE: District administrative data. 


Exhibit J.6. The Role of Magnet Conversion in the Concentration of Students From 
Minority Racial/Ethnic Backgrounds in Destination Magnet Schools 
(Average Across Schools) 


Difference Between Schools and Their Districts (%) 

Category 

Estimate 

p-value 

Change associated with conversion from 
CITS 

-1.1 

0.543 

Percentage of variation in results across 
districts that is not due to error {P) 

95.6 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: N = four schools in three districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 
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Exhibit J.7. Plot for Meta-analysis of the Role of Magnet Conversion in the 

Concentration of Students From Minority Racial/Ethnic Backgrounds in 
Destination Magnet Schools (Average Across Schools) 
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NOTE: N = three districts. 

SOURCE: District administrative data. 
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Supporting Exhibits for Research Question 2 - Concentration of 
Students From Economicaiiy Disadvantaged Backgrounds 


Exhibit J.8. Concentration of Students From Economically Disadvantaged Students in 
the Destination Magnet Schools (Average Across Schools) 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet school (%) 

25.0 (t) 

31.8 (t) 

6.8* (0.005) 

District (%) 

38.1 (t) 

41.1 (t) 

3.0* (0.033) 

Difference (%) 

13.1* (0.000) 

9.3* (0.000) 

-3.8* (0.019) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = four schools in three districts. 

SOURCE: District administrative data. 


Exhibit J.9. Percentage of Students From Economically Disadvantaged Backgrounds 
for the Magnet School and Other Neighborhood Schools in the Year Before 
Conversion, by Destination Magnet School 



Magnet Schools 

Other Neighborhood Schools (Combined) 

wGIlOOl 

Percentage 

Standard Deviation 

Percentage 

Standard Deviation 

B001 

46.1 

50.0 

45.7 

49.8 

B004 

8.6 

27.9 

43.7 

49.5 

E001 

33.7 

46.9 

51.5 

49.1 

F001 

11.5 

31.0 

11.6 

31.3 


NOTE: N = four schools in three districts. 
SOURCE: District administrative data. 


Exhibit J.10. The Role of Magnet Conversion in the Concentration of Economically 

Disadvantaged Students in Destination Magnet Schools (Average Across 
Schools) 


Difference Between Schools and Their Districts (%) 

Component 

Estimate 

p-value 

Change associated with conversion from 
CITS 

-5.7 

0.066 

Percentage of variation in results across 
districts that is not due to error (P) 

97.0 

(t) 

Test of heterogeneity 

(t) 

0.000 


* = statistically significant (p < 0.05). f = Not applicable; category does not exist. 
NOTE: N = four schools in three districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 
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Exhibit J.11. Plot for Meta-analysis of the Role of Magnet Conversion in the 

Concentration of Economically Disadvantaged Students in Destination 
Magnet Schools (Average Across Schools) 


B 


-3.85 [95% Cl: -10.46,2. 


E 


-9.98 [95% Cl: -10.91,-9 


F 

♦ 


-2.62 [95% Cl: -4.14,-11 


Overall 

4 


-12 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 


Percentage Point Difference 


-5.68 [95% Cl: -11.74,0. 


NOTE: N = three districts. 

SOURCE: District administrative data. 
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Supporting Exhibits for Research Question 3 

Exhibit J.12. Achievement in Destination Magnet Schools and Their Districts (Average 
Across Schools) 


a. ELA Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet school 

58.8 (t) 

60.2 (t) 

1.4 (0.502) 

District 

51.6 (t) 

58.5 (t) 

6.9* (0.001) 

Difference 

7.2* (0.000) 

1.7 (0.064) 

-5.5* (0.004) 


b. Mathematics Percentile Rank 


Population 

Preconversion 
Estimate (p-value) 

Postconversion 
Estimate (p-value) 

Change 

Estimate (p-value) 

Magnet school 

58.4 (t) 

62.2 (t) 

3.9 (0.189) 

District 

51.6 (t) 

60.5 (t) 

8.9* (0.002) 

Difference 

6.8* (0.004) 

1.7 (0.181) 

-5.1 (0.051) 


* = statistically significant (p < 0.05). f = Not applicable; statistical testing was not conducted for this estimate. 

NOTE: Differences were calculated using unrounded numbers and may not equal differences calculated from numbers shown in the 
table. N = four schools in three districts. 

SOURCE: District administrative data. 


Postconversion Trend Analysis for Chapter 6 

Exhibit J.13. Trend Over the Postconversion Period for the Role of Magnet Conversion 
in Changes in Destination Magnet Schools 


CITS Model 

Trend 

Estimate (p-value) 

Minority (percentage point) 

-0.08 (0.453) 

Disadvantaged (percentage point) 

-1.84* (0.046) 


* = statistically significant (p < 0.05). 

NOTE: N = four schools in three districts. CITS = comparative interrupted time series. 
SOURCE: District administrative data. 
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Appendix K: Analysis Methods and Supporting Exhibits for 
Chapter 7 

This appendix provides details on the methods and displays used for 
the analysis presented in Chapter 7 examining the variability across 
study schools in the estimated role of conversion in the change in 
achievement and diversity. 

K.1. The Analysis Models 

For results presented in Chapters 5 and 6, the CITS models — equation F.3 for the diversity CITS 
analysis for research question 2 and equation F.6 for the achievement CITS analysis for research 
question 3 — produced combined estimates across all magnet schools within a district when there 
was more than one magnet school in the district. Hence, the meta-analysis was conducted across 
district results rather than across school results. As explained in Appendix F.2.2.3, the rationale 
for combining estimates is that separate estimates for study magnet schools operating in the same 
district may be correlated and this could lead to a false sense of precision. 

Because the focus in Chapter 7 is on variability in results across magnet schools, equations F.3 
and F.6 were modified to allow for separate estimates for each study magnet school in instances 
where multiple study magnet schools were located in the same district. For the school-by-school 
analysis of the absolute value of the difference (AVD) in the proportion of students from 
minority raciahethnic backgrounds and disadvantaged backgrounds, equation F.3 was modified 
to the following; 

= ELi + Tl2F3t + Tl3F4t + Tl4F5t + TTsFgt + 'Lm=-^ Qm^REATs=m.t + ^st 


(K.1) 


where: 

■ m = indexes magnet schools in the district 

■ TREATi=m,r = (1 if school s is an MSAP school m in year t when the program was active; 
0 otherwise) 

The coefficient on TREAT, 0^, represents the association between magnet school conversion 
and the change in the absolute value of the difference, AVD, between the postconversion and 
preconversion periods for study magnet school m. As with equation F.3, equation K.l was 
estimated separately for each district. 


The number of magnet schools in each district and their classification as traditional or destination can be seen in 
Exhibit A.6. Six of the 10 study districts with at least one traditional magnet school contained more than one. One of 
the three study districts with at least one destination magnet school contained more than one. 
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Similarly, for the CITS analysis of annual achievement gains, equation F.6 was modified to the 
following: 




y 

'q^qigt 


+ 1^=2 + 


Y.U + iLi <PsNONig,t + iLi <PsUNKig,t + 


^iFzt + + T^iPst + T^Spbt + 


YJm=l{PNBH,m NBHs=m.tTREATtgst] + YJm=l{PNON,m NONs^yjif-TREATigsf-] + 
TAm=l{^UNK,m UNK.^^^tTREATig,,] + 

^igst (K.2) 


The coefficient on TREAT, Qnbh.m, represents the average change in annual achievement gain 
for neighborhood students associated with being enrolled in study magnet school m. As with 
equation F.6, equation K.2 was estimated separately for each district. 

The tests of variation across schools reported in Chapter 7 were obtained from a meta-analysis of 
the school-level results. The methods used for the meta- analysis of these school-level results 
were the same as the methods used for district-level results described in Appendix F.2.2.3. The 
results of tests of variation across schools, reported below in Exhibits K.6 and K.7, were similar 
to tests of variation across districts reported in Appendices I and J (see Exhibits 1.6, 1. 10, 1.14, 
and 1.17 for traditional schools and Exhibits J.6 and J.IO for destination schools). 

K.2. Exhibits 

The first exhibit in Chapter 7 — Exhibit 7.1 — reports school-by-school CITS results for the 
concentration of racial/ethnic minority students (AVDm) and of economically disadvantaged 
students (AVDd) for traditional magnet schools, based on the estimated coefficients 0m from 
equation K.l. For example, for the first school in Exhibit 7.1 (reprinted as Exhibit K.l), the bar 
on the left, demonstrates the magnitude of the estimated change in AVDm for that school 
associated with magnet conversion. For this school, conversion was associated with a significant 
drop in AVDm (-2.5 percentage points). The bar on the right demonstrates the magnitude of the 
estimated change in AVDd associated with magnet conversion (-2.4 percentage points). The 
asterisk (*) at the end of the bars indicates that the magnitudes were statistically significant 
(p<0.05). 
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Exhibit. K.1. Reprint for First School in Exhibit 7.1. Change in Diversity Associated With 
Magnet School Conversion 
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Exhibit 7.2 reports school-by-school CITS results for the average annual ELA achievement gain 
and the average annual mathematics achievement gain associated with magnet conversion, based 
on the estimated coefficient Bneh.ih from equation K.2. Although equation K.2 was estimated 
using z-scores, results in Exhibit K.2 are presented in terms of the annual percentile gain, using 
a process similar to that described in section E.4.2.2. As an example, results for the first school 
presented in Exhibit 7.2 are reprinted in Exhibit K.2 with the estimated value of QNBH.m for 
ELA achievement gain represented by the bar on the left and for mathematics achievement gain 
represented by the bar on the right. As in Exhibit K.l, the asterisk (*) at the end of the bars 
indicates that the magnitudes were statistically significant (p < 0.05). 

The z-score gain results from the estimation of equation K.2 were converted to percentile points 
in a manner similar to conversion of meta-analysis achievement described in section F.5.2. Using 
the ELA estimate to illustrate this, the process was as follows: 

■ First, the average postconversion ELA z-score gain for the school was computed by 
averaging across the four postconversion years. That value was 0.06. We then 
transformed this into percentile units using equation F.9. That value was 2.5. 

■ Second, the average postconversion ELA z-score gain if the school had not converted was 
estimated by subtracting the CITS estimate for the school, 0.13, from the average 
postconversion z-score gain (0.06 - 0.13 = -0.07). This gain score was then transformed 
into percentile units using equation F.9. That value was -2.8. 

■ Finally, the CITS estimate for the school in terms of a percentile point gain was 
calculated by computing the difference between the average postconversion percentile 
gain and the postconversion percentile gain if the school had not converted. That value 
was 5.3 (= 2.5 - [-2.8]), and is represented by the bar on the left of Exhibit K.2. The 
asterisk (*) at the end of the bar indicates that the magnitude was statistically significant 
(p<0.05). 
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Exhibit. K.2. Reprint for School A001 of Exhibit 7.2. Change in Learning Associated With 
Magnet School Conversion, for Neighborhood Students 
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Exhibit 7.3 was constructed in the same way as Exhibit 7.1 but with results for destination magnet 
schools. 

K.3. Supporting Exhibits 


Exhibit K.3. Number of Traditional Magnet Schools With a Significant Change in 

Diversity Associated With Magnet School Conversion, by Outcome Variable 


Outcome 

Significant 

Decrease^ 

No Significant 
Change 

Significant 

increase 

Difference between the magnet school and the 
district in proportion minority 

9 

5 

3 

Difference between the magnet school and the 
district in proportion disadvantaged 

6 

3 

2 


® A decrease is expected based on the theory of action. 

NOTE: N = 17 study magnet schools in 10 districts for the proportion minority difference from the district; N = 1 1 study magnet 
schools in seven districts for the proportion disadvantaged difference from the district. Statistical significance was determined using 
the criterion p < .05 using a two-tailed test. 

SOURCE: District administrative data. 
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Exhibit K.4. Number of Traditional Magnet Schools With a Significant Change in 
Learning for Neighborhood Students Associated With Magnet School 
Conversion, by Outcome Variable 


Outcome 

Significant 

Decrease 

No Significant 
Change 

Significant 

increase^ 

ELA annual gain for neighborhood students 

7 

5 

5 

Mathematics annual gain for neighborhood students 

6 

2 

9 


® An increase is expected based on the theory of action. 

NOTE: N = 17 study magnet schoois in 10 districts. The dependent variabie was the annuai change in a student’s achievement 
percentiie. The oniy statisticai test conducted for numbers in this exhibit was whether the “change associated with conversion” was 
significantiy different from zero. Statisticai significance was determined using the criterion p < .05 using a two-taiied test. 

SOURCE: District administrative data. 


Exhibit K.5. Number of Destination Magnet Schools With a Significant Change in 

Diversity Associated With Magnet School Conversion, by Outcome Variable 


Outcome 

Significant 

Decrease^ 

No Significant 
Change 

Significant 

increase 

Difference between the magnet school and the 
district in proportion minority 

1 

2 

1 

Difference between the magnet school and the 
district in proportion disadvantaged 

3 

1 

0 


® A decrease is expected based on the theory of action. 


NOTE: N = four study magnet schoois in three districts. The anaiysis for this dispiay was conducted using the annuai absoiute vaiue 
of the difference between each schooi and its district. Statisticai significance was determined using the criterion p < .05 using a two- 
taiied test. 

SOURCE: District administrative data. 


Exhibit K.6. Variation in Results Across Traditional Magnet Schools 


Percentage of Variation in Resuits Across Schoois That is Not Due to Error 

CiTS Modei 

P 

p-vaiue for Test of 
Heterogeneity 

Minority 

94.8 

0.000 

Disadvantaged 

96.4 

0.000 

ELA 

98.6 

0.000 

Mathematics 

98.3 

0.000 


SOURCE: District administrative data. 


Exhibit K.7. Variation in Results Across Destination Magnet Schools 


Percentage of Variation in Resuits Across Schoois That is Not Due to Error 

CiTS Modei 

P 

p-vaiue for Test of 
Heterogeneity 

Minority 

94.5 

0.000 

Disadvantaged 

96.8 

0.000 


SOURCE: District administrative data. 
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